Systems and methods for screenshot linking

ABSTRACT

A system for analyzing screenshots can include a computing device including a processor coupled to a memory and a display screen configured to display content. The system can include an application stored on the memory and executable by the processor. The application can include a screenshot receiver configured to access, from storage to which a screenshot of the content displayed on the display screen captured using a screenshot function of the computing device is stored, the screenshot including an image and a predetermined marker. The application can include a marker detector configured to detect the predetermined marker included in the screenshot. The application can include a link identifier configured to identify, using the predetermined marker, a link to a resource mapped to the image included in the screenshot, the resource accessible by the computing device via the link.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of priorityto U.S. Nonprovisional patent application Ser. No. 16/989,573, filedAug. 10, 2020, which is a continuation of U.S. Nonprovisional patentapplication Ser. No. 15/344,342, filed Nov. 4, 2016, now U.S. Pat. No.10,740,387, issued Aug. 11, 2020, which is a continuation of U.S.Nonprovisional patent application Ser. No. 15/211,321, filed Jul. 15,2016, now U.S. Pat. No. 10,423,657, issued Sep. 24, 2019, which claimsthe benefit of priority to U.S. Provisional Patent Application No.62/192,873, filed Jul. 15, 2015, all of which are incorporated herein byreference in their entireties.

FIELD OF THE DISCLOSURE

The present disclosure is generally directed to linking screenshots toother resources. In particular, the present disclosure describestechniques for analyzing images to identify a screenshot and provide alink to another resource or content associated with the screenshot.

BACKGROUND OF THE DISCLOSURE

Companies can use the Internet to advertise or promote their brands andproducts. Users of client devices can view content provided by companiesusing a web browser or other application. Some users may capturescreenshots relating to content that interests them on their clientdevices.

BRIEF SUMMARY OF THE DISCLOSURE

Systems and methods of the present solution are directed to linkingscreenshots to other resources. Consumer products companies oftenproduce content that is published via the Internet. For example, acompany may have its own website including web pages dedicated to itsproducts or to brands associated with the company. In some instances,companies may promote their brands and products through third party webpages, such as social media sites. For example, many companies have oneor more accounts on websites such as Facebook, Instagram, Pinterest, andTwitter. Users can view content published by a company by visiting thecompany's web pages or third party web pages on which the companypublishes content. In some cases, users also may view content publishedby a company using a dedicated application on a client device, such asan application associated with a social media website on which thecompany publishes content.

Users can save to their devices images of content that interests them onthe Internet. Users can accomplish this by capturing a screenshot of adisplay of the client device while the client device displays selectedcontent. Typically, a screenshot refers to an image representing aportion or the entire graphical user interface of a display of a clientdevice. Thus, a screenshot captured while the client device is beingused to display content published by a company (for example, in a webbrowser or a third party application), will include the relevant contentthat the user wishes to save, and may also include other elements of thegraphical user interface of the client device. A screenshot is oftensaved as an image file on the client device. In some client devices, thescreenshots are saved in an image folder of the client device. The imagefolder is capable of storing the screenshots captured by the clientdevice, regardless of the content or the application displayed on theclient device at the time the screenshot was captured. In this way, theimage folder can store screenshots of content displayed by multipledifferent applications of the client device.

Because screenshots typically include only image data, it can bedifficult for a user to access content relevant to what is depicted in ascreenshot when the user only has access to the screenshot itself. Forexample, while the screenshot may provide an image of a product that theuser would like to purchase, the image file associated with thescreenshot itself does not provide any interface with which the user caninteract to purchase the product shown in the screenshot. The presentdisclosure provides techniques for linking a screenshot to otherrelevant resources and/or content, such as a shopping interface.

One aspect of the present disclosure can be implemented in a method orsystem for providing an online store associated with a screenshot. Ascreenshot linking system including one or more processors can identifya screenshot from images stored on a device, such as a photo roll on amobile device storing both photos and screenshots. The screenshotlinking system can identify an image included within the screenshot. Thescreenshot linking system can match the identified image with areference image of a plurality of reference images maintained by thescreenshot linking system. Each reference image can be associated with alink to content. In some implementations, the link can be a link tocontent on a mobile application or content on a webpage. In someimplementations, the content to which the reference image is linked caninclude the reference image. Responsive to the screenshot linking systemmatching the image with a reference image, the screenshot linking systemcan then identify a link associated with the image. In someimplementations, the screenshot linking system can generate and providea notification to the device associated with the screenshot. Thenotification can include the link to content corresponding to the imageidentified in the screenshot. In some implementations, if the image inthe screenshot is an image of a product, the link provided in thenotification can be a link to a page of a mobile application or a webserver through which the user of the device can purchase or otherwisereceive additional information about the product. For example, if theimage in the screenshot is of a shirt, the screenshot linking system canidentify a link to a product page where the shirt is available forpurchase, and can provide a notification including a link to the productpage. In some implementations, responsive to the user visiting the pagevia the link, the screenshot linking system can receive, from thedevice, a request to purchase the product and execute the request topurchase the product. In some implementations, it can be determined thatan image file corresponds to a screenshot, and the screenshot can beanalyzed to identify at least one application associated with thescreenshot.

As discussed above, users of computing devices may wish to save imagesof content that interests them, such as from web pages on the Internetor screens of mobile applications, in the form of screenshots. Ascreenshot can be processed to generate a link to other relevantcontent. In some implementations, it may be useful to determine anapplication associated with a screenshot for purposes of generating sucha link. For example, a deep link may be a link to a particular page orportion of a particular application, such as the application that wasused to generate the screenshot. Therefore, for the screenshot linkingsystem to generate a deep link, the screenshot linking system mayidentify the identity of the application active on the display of theclient device at the time the screenshot was captured. Furthermore, thescreenshot linking system may also be useful to process only a subset ofthe screenshots saved on a client device, such as screenshots that aregenerated by only a subset of applications. For example, someapplications may be added to a white list of applications whosescreenshots are to be processed and linked to other content. Screenshotsassociated with applications not included in the white list may not beprocessed in order to save computing resources.

One aspect of this disclosure can be implemented in a method or systemfor determining an identity of an application that is associated with aparticular screenshot. The screenshot linking system can identify animage file and determine that the image file corresponds to ascreenshot. The screenshot linking system can identify one or morevisual elements of the screenshot. The screenshot linking system canthen determine an identity of an application corresponding to thescreenshot based on the identified visual elements.

As discussed above, a screenshot may capture the graphical contentdisplayed on a display screen of a client device. However, some of thevisual elements of the screenshot may be irrelevant, such as elements ofa user interface of the active application or other information that maybe displayed on a client device, such as the time, date, and batterylevel of the client device. Therefore, it can be useful to separate therelevant content of a screenshot, such as an image provided by a contentpublisher, from the less relevant content that may be included withinthe screenshot. After an image has been identified and/or extracted, theimage may be processed to determine a watermark or fingerprintassociated with the image. Watermarks and fingerprints can be used tomatch the images with reference images provided by content publishers.In this way, if a user takes a screenshot of an image that matches areference image of a content publisher, the screenshot linking systemcan provide the user, via the device of the user, a notification thatincludes a link to content corresponding to the reference image thatmatched the image of the screenshot. As a result, content publishers oradvertisers can provide relevant content to users based on screenshotscaptured by the users.

One aspect of this disclosure can be implemented in a method or systemfor analyzing an image included within a screenshot. An image can beidentified and/or extracted from a screenshot. A watermark of the imagecan be identified. The watermark can include additional informationcorresponding to the image. In some implementations, the watermark caninclude data representing a link to a page of a mobile application orweb server that includes content associated with the image. The link canbe a link to content of a product shown in the image. In someimplementations, the link can be a link to content of an advertiser orentity that published or provided the image. In some implementations,responsive to identifying the link to content, the screenshot linkingsystem can provide a notification to the client device including thelink to content to allow the user of the client device to retrieveadditional information relating to the image included in the screenshot.

Another aspect of this disclosure also can be implemented in a methodfor analyzing an image included within a screenshot. An image can beidentified and/or extracted from a screenshot. The screenshot linkingsystem or a mobile application corresponding to the screenshot linkingsystem can generate a fingerprint of the image can be generated. Thescreenshot linking system can then identify a fingerprint of a referenceimage that matches the fingerprint of the image of the screenshot. Thescreenshot linking system, responsive to determining that thefingerprint of the image of the screenshot matches the identifiedfingerprint of the reference image, can identify a link to contentassociated with the image. The screenshot linking system can thengenerate and provide a notification including the link to the user ofthe device. In this way, the screenshot linking system can provide theuser relevant content based on images included in screenshots capturedby the user. As a result, advertisers or content publishers can sendlinks relevant to the user to increase user engagement.

Users who capture screenshots depicting particular types of products maynot be interested in other unrelated types of products. However,products similar to those that are depicted in the screenshots that auser captures may be of more interest to the user. As a result, the usermay be more likely to purchase a product that is similar to the productsdepicted in the screenshots captured by the user. In someimplementations, the screenshots and purchasing decisions of the user,as well as those of other users, can form the basis for a productrecommendation for the user.

One aspect of this disclosure can be implemented in a method forproviding a product recommendation to a user. To provide recommendationto one or more users, the screenshot linking system can build one ormore recommendation engines. The recommendation engines can rely onmodels that process input data to generate output data in the form ofrecommendations. The input data can be referred to as training data thatis provided to the recommendation engine to train the model. The modelscan be adaptive and continually learning.

In some implementations, the recommendation engine can be configured todetermine recommendations for a user based on a user's screenshots. Therecommendation engine can determine recommendations of products that aresimilar to products identified in the user's screenshots. In someimplementations, the recommendation engine can rely on screenshots ofother users that the recommendation engine determines are similar toscreenshots captured by the user. In this way, the recommendation enginecan generate product recommendations for the user based on screenshotsand images included in the screenshots captured by the other users ofthe screenshot linking system.

In some implementations, the recommendation engine can be configured toidentify users that are similar to one another based on the screenshotscaptured by each of the users. In this way, the recommendation enginecan recommend products in which users similar to the user have expressedan interest. Similarly, advertisers or other content providers maychoose to generate marketing or advertising campaigns directed towardsusers that are similar to one another and that are likely interested inthe products of the advertisers. The recommendation engine can beconfigured to generate a mode to identify users similar to one anotherbased on the screenshots of each of the users.

According to one aspect, a first plurality of user profiles can begenerated for a plurality of users based on screenshots associated withthe plurality of users. A plurality of categories to which the userprofiles belong can be determined. Each user profile can be assigned toat least one of the plurality of categories. A recommendation model canbe built for each category. A second user profile, not included in thefirst plurality of user profiles, can be identified. A category to whichthe second user profile should be assigned can be identified. A productrecommendation can be provided to the second user based on the categoryto which the second user profile is assigned.

According to another aspect, the screenshot linking system can receive aplurality of screenshots captured by a device of a user. The screenshotlinking system can identify images included in the screenshot. Thescreenshot linking system can, from the images, identify one or morecategories to which the images correspond. In some implementations, thecategories can be based on topics, such as clothing, furniture, orelectronics, among others. In some implementations, the categories canbe based on brands. In some implementations, the categories can be basedon other attributes. Responsive to determining the categories to whichthe screenshots belong, the screenshot linking system can generate auser profile based on the categories of the screenshots. The screenshotlinking system, via a recommendation engine that uses data from a largenumber of users, can provide recommendations of products (or content ingeneral) based on a comparison of the user's profile with user profilesof other users and the screenshots corresponding to the other users.

At least one aspect of this disclosure can be implemented as method foranalyzing screenshots to identify links to resources. The method caninclude accessing, by an application executing on a computing devicefrom storage to which a screenshot captured using a screenshot capturefunction of the computing device is stored, the screenshot including animage and a predetermined marker. The method can include detecting, bythe application, the predetermined marker included in the screenshot.The method can include identifying, by the application, using thepredetermined marker, a link to a resource mapped to the image includedin the screenshot. The method can include accessing the resource via thelink.

In some implementations, the storage to which the screenshot is storedcan be accessible by multiple applications executing on the computingdevice. In some implementations, the predetermined marker can include avisual marker added to the image included in the screenshot. In someimplementations, the predetermined marker can include metadata uniquelyidentifying the image included in the screenshot or the image can beencoded with data corresponding to the predetermined marker. In someimplementations, the predetermined marker can include a code comprisinga string of characters.

In some implementations, the method can include comparing thepredetermined marker to a plurality of reference markers. The methodalso can include determining a match between the predetermined markerand one of the plurality of reference markers. The method also caninclude selecting the link to the resource based on the one of theplurality of reference markers.

In some implementations, accessing the resource via the link can includeaccessing the resource responsive to selection of the link from anotification including the link. In some implementations, theapplication can be a first application and the link can include a deeplink corresponding to a resource within a second application executableon the computing device.

Another aspect of this disclosure can be implemented as system foranalyzing screenshots to identify links to resources. The system caninclude a computing device including a processor coupled to a memory anda display screen configured to display content. The system can includean application stored on the memory and executable by the processor. Theapplication can include a screenshot receiver configured to access, fromstorage to which a screenshot of the content displayed on the displayscreen captured using a screenshot function of the computing device isstored, the screenshot including an image and a predetermined marker.The application can include a marker detector configured to detect thepredetermined marker included in the screenshot. The application caninclude a link identifier configured to identify, using thepredetermined marker, a link to a resource mapped to the image includedin the screenshot, the resource accessible by the computing device viathe link.

In some implementations, the storage to which the screenshot is storedcan be accessible by multiple applications executing on the computingdevice. In some implementations, the predetermined marker can include avisual marker added to the image included in the screenshot. In someimplementations, the predetermined marker can include metadata uniquelyidentifying the image included in the screenshot or the image can beencoded within data corresponding to the predetermined marker. In someimplementations, the predetermined marker can include a code comprisinga string of characters.

In some implementations, the system also can include a screenshotlinking server. The screenshot linking server can be configured toreceive the predetermined marker from the computing device. Thescreenshot linking server can be configured to compare the predeterminedmarker to a plurality of reference markers. The screenshot linkingserver can be configured to determine a match between the predeterminedmarker and one of the plurality of reference markers. The screenshotlinking server can be configured to select the link based on the one ofthe plurality of reference markers. In some implementations, theapplication can be a first application and the link can include a deeplink corresponding to a resource within a second application executableon the computing device.

Another aspect of this disclosure can be implemented as a method foranalyzing screenshots to identify links to resources. The method caninclude accessing, by an application executing on a computing devicefrom storage to which a screenshot has been captured by a screenshotfunction of the computing device, the screenshot including an image. Themethod can include extracting, by the application, data corresponding tothe image included in the screenshot. The method can include generating,by the application, a fingerprint identifying the image, the fingerprintgenerated based on the data corresponding to the image. The method caninclude identifying, by the application, a link to a resource mapped tothe fingerprint of the image. The method can include accessing theresource via the link.

In some implementations, the method can include identifying the imagefrom the screenshot by performing image entropy analysis on thescreenshot to determine a portion of the screenshot that corresponds tothe image. In some implementations, the application can be a firstapplication and the link can include a deep link corresponding to aresource within a second application that can execute on the computingdevice. In some implementations, identifying, by the application, thelink to the resource mapped to the fingerprint of the image can furtherinclude identifying the link responsive to determining a match betweenthe fingerprint and a plurality of reference fingerprints based on acomparison of the fingerprint of the image to a plurality of referencefingerprints.

Another aspect of this disclosure can be implemented as a system foranalyzing screenshots to identify links to resources. The system caninclude a computing device including a processor coupled to a memory anda display screen configured to display content. The system can includean application stored on the memory and executable by the processor. Theapplication can include a screenshot receiver configured to access, fromstorage to which a screenshot of the content displayed on the displayscreen captured using a screenshot function of the computing device isstored, the screenshot including an image. The application can includean image extractor configured to extract data corresponding to the imageincluded in the screenshot. The application can include a fingerprintgenerator configured to generate a fingerprint identifying the imageincluded in the screenshot based on the data corresponding to the image.The application can include a link identifier configured to identify,using the fingerprint, a link to a resource mapped to the image includedin the screenshot. The application can include a resource displaycomponent configured to access the resource via the link.

In some implementations, the image extractor can be further configuredto identify the image from the screenshot by performing image entropyanalysis on the screenshot to determine a portion of the screenshot thatcorresponds to the image. In some implementations, the application canbe a first application and the link can include a deep linkcorresponding to a resource within a second application that can executeon the computing device. In some implementations, the link identifiercan be configured to identify the link to the resource responsive todetermining a match between the fingerprint and a plurality of referencefingerprints based on a comparison of the fingerprint of the image to aplurality of reference fingerprints.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects, features, and advantages ofthe disclosure will become more apparent and better understood byreferring to the following description taken in conjunction with theaccompanying drawings, in which:

FIG. 1A is a block diagram depicting an embodiment of a networkenvironment comprising a client device in communication with serverdevice;

FIG. 1B is a block diagram depicting a cloud computing environmentcomprising client device in communication with cloud service providers;

FIGS. 1C and 1D are block diagrams depicting embodiments of computingdevices useful in connection with the methods and systems describedherein;

FIG. 2A is a block diagram depicting an embodiment of a system forlinking a screenshot to other content;

FIG. 2B is a conceptual diagram depicting an example use case for thesystem of FIG. 2A;

FIGS. 2C-2G illustrate various stages of a process for linking ascreenshot to a product purchase page;

FIG. 2H is a flow chart of a process for linking a screenshot to othercontent;

FIG. 2I is a flow chart of a process for providing an online storeassociated with a screenshot;

FIG. 3A is a block diagram depicting an embodiment of a screenshotanalyzer that can be used in the system of FIG. 2A;

FIG. 3B is an example screenshot including an image;

FIG. 3C is an example graphical user interface indicating an applicationassociated with a screenshot;

FIG. 3D is flow chart of a process for determining an applicationassociated with a screenshot;

FIG. 4A is a block diagram depicting an embodiment of an image analyzerthat can be used in the system of FIG. 2A;

FIG. 4B is a conceptual diagram depicting an example use case for thesystem of FIG. 2A;

FIG. 4C is a conceptual diagram depicting an example use case for thesystem of FIG. 2A;

FIG. 4D is a flow chart of a process for analyzing an image including awatermark;

FIG. 4E is a flow chart of a process for analyzing an image based on afingerprint of the image;

FIG. 4F is a flow chart of a process for analyzing screenshots toidentify links to resources;

FIG. 5A is a block diagram depicting an embodiment of a contentrecommendation provider that can be used in the system of FIG. 2A;

FIG. 5B is an example graphical user interface for displaying productsfound in the screenshots of a client device;

FIG. 5C is an example graphical user interface for displaying productssimilar to those that are found in the screenshots of a client device;

FIG. 5D is a graphical user interface for displaying the screenshotsthat are stored on a client device; and

FIG. 5E is a flow chart of a method for providing a productrecommendation to a user.

DETAILED DESCRIPTION

For purposes of reading the description of the various embodimentsbelow, the following descriptions of the sections of the specificationand their respective contents may be helpful:

Section A describes a network environment and computing environmentwhich may be useful for practicing embodiments described herein.

Section B describes embodiments of systems and methods for linking ascreenshot to other content.

Section C describes embodiments of systems and methods for determiningan application associated with a screenshot.

Section D describes embodiments of systems and methods for analyzing animage included within a screenshot.

Section E describes embodiments of systems and methods for providing aproduct recommendation to a user.

A. Computing and Network Environment

Prior to discussing specific embodiments of the present solution, it maybe helpful to describe aspects of the operating environment as well asassociated system components (e.g., hardware elements) in connectionwith the methods and systems described herein. Referring to FIG. 1A, anembodiment of a network environment is depicted. In brief overview, thenetwork environment includes one or more clients 102 a-102 n (alsogenerally referred to as local machine(s) 102, client(s) 102, clientnode(s) 102, client machine(s) 102, client computer(s) 102, clientdevice(s) 102, endpoint(s) 102, or endpoint node(s) 102) incommunication with one or more servers 106 a-106 n (also generallyreferred to as server(s) 106, node 106, or remote machine(s) 106) viaone or more networks 104. In some embodiments, a client 102 has thecapacity to function as both a client node seeking access to resourcesprovided by a server and as a server providing access to hostedresources for other clients 102 a-102 n.

Although FIG. 1A shows a network 104 between the clients 102 and theservers 106, the clients 102 and the servers 106 may be on the samenetwork 104. In some embodiments, there are multiple networks 104between the clients 102 and the servers 106. In one of theseembodiments, a network 104′ (not shown) may be a private network and anetwork 104 may be a public network. In another of these embodiments, anetwork 104 may be a private network and a network 104′ a publicnetwork. In still another of these embodiments, networks 104 and 104′may both be private networks.

The network 104 may be connected via wired or wireless links. Wiredlinks may include Digital Subscriber Line (DSL), coaxial cable lines, oroptical fiber lines. The wireless links may include BLUETOOTH, Wi-Fi,Worldwide Interoperability for Microwave Access (WiMAX), an infraredchannel or satellite band. The wireless links may also include anycellular network standards used to communicate among mobile devices,including standards that qualify as 1G, 2G, 3G, or 4G. The networkstandards may qualify as one or more generation of mobiletelecommunication standards by fulfilling a specification or standardssuch as the specifications maintained by International TelecommunicationUnion. The 3G standards, for example, may correspond to theInternational Mobile Telecommunications-2000 (IMT-2000) specification,and the 4G standards may correspond to the International MobileTelecommunications Advanced (IMT-Advanced) specification. Examples ofcellular network standards include AMPS, GSM, GPRS, UMTS, LTE, LTEAdvanced, Mobile WiMAX, and WiMAX-Advanced. Cellular network standardsmay use various channel access methods e.g. FDMA, TDMA, CDMA, or SDMA.In some embodiments, different types of data may be transmitted viadifferent links and standards. In other embodiments, the same types ofdata may be transmitted via different links and standards.

The network 104 may be any type and/or form of network. The geographicalscope of the network 104 may vary widely and the network 104 can be abody area network (BAN), a personal area network (PAN), a local-areanetwork (LAN), e.g. Intranet, a metropolitan area network (MAN), a widearea network (WAN), or the Internet. The topology of the network 104 maybe of any form and may include, e.g., any of the following:point-to-point, bus, star, ring, mesh, or tree. The network 104 may bean overlay network which is virtual and sits on top of one or morelayers of other networks 104′. The network 104 may be of any suchnetwork topology as known to those ordinarily skilled in the art capableof supporting the operations described herein. The network 104 mayutilize different techniques and layers or stacks of protocols,including, e.g., the Ethernet protocol, the internet protocol suite(TCP/IP), the ATM (Asynchronous Transfer Mode) technique, the SONET(Synchronous Optical Networking) protocol, or the SDH (SynchronousDigital Hierarchy) protocol. The TCP/IP internet protocol suite mayinclude application layer, transport layer, internet layer (including,e.g., IPv6), or the link layer. The network 104 may be a type of abroadcast network, a telecommunications network, a data communicationnetwork, or a computer network.

In some embodiments, the system may include multiple, logically-groupedservers 106. In one of these embodiments, the logical group of serversmay be referred to as a server farm 38 or a machine farm 38. In anotherof these embodiments, the servers 106 may be geographically dispersed.In other embodiments, a machine farm 38 may be administered as a singleentity. In still other embodiments, the machine farm 38 includes aplurality of machine farms 38. The servers 106 within each machine farm38 can be heterogeneous—one or more of the servers 106 or machines 106can operate according to one type of operating system platform (e.g.,WINDOWS NT, manufactured by Microsoft Corp. of Redmond, Wash.), whileone or more of the other servers 106 can operate on according to anothertype of operating system platform (e.g., Unix, Linux, or Mac OS X).

In one embodiment, servers 106 in the machine farm 38 may be stored inhigh-density rack systems, along with associated storage systems, andlocated in an enterprise data center. In this embodiment, consolidatingthe servers 106 in this way may improve system manageability, datasecurity, the physical security of the system, and system performance bylocating servers 106 and high performance storage systems on localizedhigh performance networks. Centralizing the servers 106 and storagesystems and coupling them with advanced system management tools allowsmore efficient use of server resources.

The servers 106 of each machine farm 38 do not need to be physicallyproximate to another server 106 in the same machine farm 38. Thus, thegroup of servers 106 logically grouped as a machine farm 38 may beinterconnected using a wide-area network (WAN) connection or ametropolitan-area network (MAN) connection. For example, a machine farm38 may include servers 106 physically located in different continents ordifferent regions of a continent, country, state, city, campus, or room.Data transmission speeds between servers 106 in the machine farm 38 canbe increased if the servers 106 are connected using a local-area network(LAN) connection or some form of direct connection. Additionally, aheterogeneous machine farm 38 may include one or more servers 106operating according to a type of operating system, while one or moreother servers 106 execute one or more types of hypervisors rather thanoperating systems. In these embodiments, hypervisors may be used toemulate virtual hardware, partition physical hardware, virtualizephysical hardware, and execute virtual machines that provide access tocomputing environments, allowing multiple operating systems to runconcurrently on a host computer. Native hypervisors may run directly onthe host computer. Hypervisors may include VMware ESX/ESXi, manufacturedby VMWare, Inc., of Palo Alto, Calif.; the Xen hypervisor, an opensource product whose development is overseen by Citrix Systems, Inc.;the HYPER-V hypervisors provided by Microsoft or others. Hostedhypervisors may run within an operating system on a second softwarelevel. Examples of hosted hypervisors may include VMware Workstation andVIRTUALBOX.

Management of the machine farm 38 may be de-centralized. For example,one or more servers 106 may comprise components, subsystems and modulesto support one or more management services for the machine farm 38. Inone of these embodiments, one or more servers 106 provide functionalityfor management of dynamic data, including techniques for handlingfailover, data replication, and increasing the robustness of the machinefarm 38. Each server 106 may communicate with a persistent store and, insome embodiments, with a dynamic store.

Server 106 may be a file server, application server, web server, proxyserver, appliance, network appliance, gateway, gateway server,virtualization server, deployment server, SSL VPN server, or firewall.In one embodiment, the server 106 may be referred to as a remote machineor a node. In another embodiment, a plurality of nodes 290 may be in thepath between any two communicating servers.

Referring to FIG. 1B, a cloud computing environment is depicted. A cloudcomputing environment may provide client 102 with one or more resourcesprovided by a network environment. The cloud computing environment mayinclude one or more clients 102 a-102 n, in communication with the cloud108 over one or more networks 104. Clients 102 may include, e.g., thickclients, thin clients, and zero clients. A thick client may provide atleast some functionality even when disconnected from the cloud 108 orservers 106. A thin client or a zero client may depend on the connectionto the cloud 108 or server 106 to provide functionality. A zero clientmay depend on the cloud 108 or other networks 104 or servers 106 toretrieve operating system data for the client device. The cloud 108 mayinclude back end platforms, e.g., servers 106, storage, server farms ordata centers.

The cloud 108 may be public, private, or hybrid. Public clouds mayinclude public servers 106 that are maintained by third parties to theclients 102 or the owners of the clients. The servers 106 may be locatedoff-site in remote geographical locations as disclosed above orotherwise. Public clouds may be connected to the servers 106 over apublic network. Private clouds may include private servers 106 that arephysically maintained by clients 102 or owners of clients. Privateclouds may be connected to the servers 106 over a private network 104.Hybrid clouds 108 may include both the private and public networks 104and servers 106.

The cloud 108 may also include a cloud based delivery, e.g. Software asa Service (SaaS) 110, Platform as a Service (PaaS) 112, andInfrastructure as a Service (IaaS) 114. IaaS may refer to a user rentingthe use of infrastructure resources that are needed during a specifiedtime period. IaaS providers may offer storage, networking, servers orvirtualization resources from large pools, allowing the users to quicklyscale up by accessing more resources as needed. Examples of IaaS caninclude infrastructure and services (e.g., EG-32) provided by OVHHOSTING of Montreal, Quebec, Canada, AMAZON WEB SERVICES provided byAmazon.com, Inc., of Seattle, Wash., RACKSPACE CLOUD provided byRackspace US, Inc., of San Antonio, Tex., Google Compute Engine providedby Google Inc. of Mountain View, Calif., or RIGHTSCALE provided byRightScale, Inc., of Santa Barbara, Calif. PaaS providers may offerfunctionality provided by IaaS, including, e.g., storage, networking,servers or virtualization, as well as additional resources such as,e.g., the operating system, middleware, or runtime resources. Examplesof PaaS include WINDOWS AZURE provided by Microsoft Corporation ofRedmond, Wash., Google App Engine provided by Google Inc., and HEROKUprovided by Heroku, Inc. of San Francisco, Calif. SaaS providers mayoffer the resources that PaaS provides, including storage, networking,servers, virtualization, operating system, middleware, or runtimeresources. In some embodiments, SaaS providers may offer additionalresources including, e.g., data and application resources. Examples ofSaaS include GOOGLE APPS provided by Google Inc., SALESFORCE provided bySalesforce.com Inc. of San Francisco, Calif., or OFFICE 365 provided byMicrosoft Corporation. Examples of SaaS may also include data storageproviders, e.g. DROPBOX provided by Dropbox, Inc. of San Francisco,Calif., Microsoft SKYDRIVE provided by Microsoft Corporation, GoogleDrive provided by Google Inc., or Apple ICLOUD provided by Apple Inc. ofCupertino, Calif.

Clients 102 may access IaaS resources with one or more IaaS standards,including, e.g., Amazon Elastic Compute Cloud (EC2), Open CloudComputing Interface (OCCI), Cloud Infrastructure Management Interface(CIMI), or OpenStack standards. Some IaaS standards may allow clientsaccess to resources over HTTP, and may use Representational StateTransfer (REST) protocol or Simple Object Access Protocol (SOAP).Clients 102 may access PaaS resources with different PaaS interfaces.Some PaaS interfaces use HTTP packages, standard Java APIs, JavaMailAPI, Java Data Objects (JDO), Java Persistence API (JPA), Python APIs,web integration APIs for different programming languages including,e.g., Rack for Ruby, WSGI for Python, or PSGI for Perl, or other APIsthat may be built on REST, HTTP, XML, or other protocols. Clients 102may access SaaS resources through the use of web-based user interfaces,provided by a web browser (e.g. GOOGLE CHROME, Microsoft INTERNETEXPLORER, or Mozilla Firefox provided by Mozilla Foundation of MountainView, Calif.). Clients 102 may also access SaaS resources throughsmartphone or tablet applications, including, e.g., Salesforce SalesCloud, or Google Drive app. Clients 102 may also access SaaS resourcesthrough the client operating system, including, e.g., Windows filesystem for DROPBOX.

In some embodiments, access to IaaS, PaaS, or SaaS resources may beauthenticated. For example, a server or authentication server mayauthenticate a user via security certificates, HTTPS, or API keys. APIkeys may include various encryption standards such as, e.g., AdvancedEncryption Standard (AES). Data resources may be sent over TransportLayer Security (TLS) or Secure Sockets Layer (SSL).

The client 102 and server 106 may be deployed as and/or executed on anytype and form of computing device, e.g. a computer, network device orappliance capable of communicating on any type and form of network andperforming the operations described herein. FIGS. 1C and 1D depict blockdiagrams of a computing device 100 useful for practicing an embodimentof the client 102 or a server 106. As shown in FIGS. 1C and 1D, eachcomputing device 100 includes a central processing unit 121, and a mainmemory unit 122. As shown in FIG. 1C, a computing device 100 may includea storage device 128, an installation device 116, a network interface118, an I/O controller 123, display devices 124 a-124 n, a keyboard 126and a pointing device 127, e.g. a mouse. The storage device 128 mayinclude, without limitation, an operating system, software, and asoftware of a screenshot linking system 120. As shown in FIG. 1D, eachcomputing device 100 may also include additional optional elements, e.g.a memory port 103, a bridge 170, one or more input/output devices 130a-130 n (generally referred to using reference numeral 130), and a cachememory 140 in communication with the central processing unit 121.

The central processing unit 121 is any logic circuitry that responds toand processes instructions fetched from the main memory unit 122. Inmany embodiments, the central processing unit 121 is provided by amicroprocessor unit, e.g.: those manufactured by Intel Corporation ofMountain View, Calif.; those manufactured by Motorola Corporation ofSchaumburg, Ill.; the ARM processor and TEGRA system on a chip (SoC)manufactured by Nvidia of Santa Clara, Calif.; the POWER7 processor,those manufactured by International Business Machines of White Plains,N.Y.; or those manufactured by Advanced Micro Devices of Sunnyvale,Calif. The computing device 100 may be based on any of these processors,or any other processor capable of operating as described herein. Thecentral processing unit 121 may utilize instruction level parallelism,thread level parallelism, different levels of cache, and multi-coreprocessors. A multi-core processor may include two or more processingunits on a single computing component. Examples of multi-core processorsinclude the AMD PHENOM IIX2, INTEL CORE i5 and INTEL CORE i7.

Main memory unit 122 may include one or more memory chips capable ofstoring data and allowing any storage location to be directly accessedby the microprocessor 121. Main memory unit 122 may be volatile andfaster than storage 128 memory. Main memory units 122 may be Dynamicrandom access memory (DRAM) or any variants, including static randomaccess memory (SRAM), Burst SRAM or SynchBurst SRAM (BSRAM), Fast PageMode DRAM (FPM DRAM), Enhanced DRAM (EDRAM), Extended Data Output RAM(EDO RAM), Extended Data Output DRAM (EDO DRAM), Burst Extended DataOutput DRAM (BEDO DRAM), Single Data Rate Synchronous DRAM (SDR SDRAM),Double Data Rate SDRAM (DDR SDRAM), Direct Rambus DRAM (DRDRAM), orExtreme Data Rate DRAM (XDR DRAM). In some embodiments, the main memory122 or the storage 128 may be non-volatile; e.g., non-volatile readaccess memory (NVRAM), flash memory non-volatile static RAM (nvSRAM),Ferroelectric RAM (FeRAM), Magnetoresistive RAM (MRAM), Phase-changememory (PRAM), conductive-bridging RAM (CBRAIVI),Silicon-Oxide-Nitride-Oxide-Silicon (SONOS), Resistive RAM (RRAM),Racetrack, Nano-RAM (NRAM), or Millipede memory. The main memory 122 maybe based on any of the above described memory chips, or any otheravailable memory chips capable of operating as described herein. In theembodiment shown in FIG. 1C, the processor 121 communicates with mainmemory 122 via a system bus 150 (described in more detail below). FIG.1D depicts an embodiment of a computing device 100 in which theprocessor communicates directly with main memory 122 via a memory port103. For example, in FIG. 1D the main memory 122 may be DRDRAM.

FIG. 1D depicts an embodiment in which the main processor 121communicates directly with cache memory 140 via a secondary bus,sometimes referred to as a backside bus. In other embodiments, the mainprocessor 121 communicates with cache memory 140 using the system bus150. Cache memory 140 typically has a faster response time than mainmemory 122 and is typically provided by SRAM, BSRAM, or EDRAM. In theembodiment shown in FIG. 1D, the processor 121 communicates with variousI/O devices 130 via a local system bus 150. Various buses may be used toconnect the central processing unit 121 to any of the I/O devices 130,including a PCI bus, a PCI-X bus, or a PCI-Express bus, or a NuBus. Forembodiments in which the I/O device is a video display 124, theprocessor 121 may use an Advanced Graphics Port (AGP) to communicatewith the display 124 or the I/O controller 123 for the display 124. FIG.1D depicts an embodiment of a computer 100 in which the main processor121 communicates directly with I/O device 130 b or other processors 121′via HYPERTRANSPORT, RAPIDIO, or INFINIBAND communications technology.FIG. 1D also depicts an embodiment in which local busses and directcommunication are mixed: the processor 121 communicates with I/O device130 a using a local interconnect bus while communicating with I/O device130 b directly.

A wide variety of I/O devices 130 a-130 n may be present in thecomputing device 100. Input devices may include keyboards, mice,trackpads, trackballs, touchpads, touch mice, multi-touch touchpads andtouch mice, microphones, multi-array microphones, drawing tablets,cameras, single-lens reflex camera (SLR), digital SLR (DSLR), CMOSsensors, accelerometers, infrared optical sensors, pressure sensors,magnetometer sensors, angular rate sensors, depth sensors, proximitysensors, ambient light sensors, gyroscopic sensors, or other sensors.Output devices may include video displays, graphical displays, speakers,headphones, inkjet printers, laser printers, and 3D printers.

Devices 130 a-130 n may include a combination of multiple input oroutput devices, including, e.g., Microsoft KINECT, Nintendo Wiimote forthe WII, Nintendo WII U GAMEPAD, or Apple IPHONE. Some devices 130 a-130n allow gesture recognition inputs through combining some of the inputsand outputs. Some devices 130 a-130 n provides for facial recognitionwhich may be utilized as an input for different purposes includingauthentication and other commands. Some devices 130 a-130 n provides forvoice recognition and inputs, including, e.g., Microsoft KINECT, SIRIfor IPHONE by Apple, Google Now or Google Voice Search.

Additional devices 130 a-130 n have both input and output capabilities,including, e.g., haptic feedback devices, touchscreen displays, ormulti-touch displays. Touchscreen, multi-touch displays, touchpads,touch mice, or other touch sensing devices may use differenttechnologies to sense touch, including, e.g., capacitive, surfacecapacitive, projected capacitive touch (PCT), in-cell capacitive,resistive, infrared, waveguide, dispersive signal touch (DST), in-celloptical, surface acoustic wave (SAW), bending wave touch (BWT), orforce-based sensing technologies. Some multi-touch devices may allow twoor more contact points with the surface, allowing advanced functionalityincluding, e.g., pinch, spread, rotate, scroll, or other gestures. Sometouchscreen devices, including, e.g., Microsoft PIXELSENSE orMulti-Touch Collaboration Wall, may have larger surfaces, such as on atable-top or on a wall, and may also interact with other electronicdevices. Some I/O devices 130 a-130 n, display devices 124 a-124 n orgroup of devices may be augment reality devices. The I/O devices may becontrolled by an I/O controller 123 as shown in FIG. 1C. The I/Ocontroller may control one or more I/O devices, such as, e.g., akeyboard 126 and a pointing device 127, e.g., a mouse or optical pen.Furthermore, an I/O device may also provide storage and/or aninstallation medium 116 for the computing device 100. In still otherembodiments, the computing device 100 may provide USB connections (notshown) to receive handheld USB storage devices. In further embodiments,an I/O device 130 may be a bridge between the system bus 150 and anexternal communication bus, e.g. a USB bus, a SCSI bus, a FireWire bus,an Ethernet bus, a Gigabit Ethernet bus, a Fibre Channel bus, or aThunderbolt bus.

In some embodiments, display devices 124 a-124 n may be connected to I/Ocontroller 123. Display devices may include, e.g., liquid crystaldisplays (LCD), thin film transistor LCD (TFT-LCD), blue phase LCD,electronic papers (e-ink) displays, flexile displays, light emittingdiode displays (LED), digital light processing (DLP) displays, liquidcrystal on silicon (LCOS) displays, organic light-emitting diode (OLED)displays, active-matrix organic light-emitting diode (AMOLED) displays,liquid crystal laser displays, time-multiplexed optical shutter (TMOS)displays, or 3D displays. Examples of 3D displays may use, e.g.stereoscopy, polarization filters, active shutters, or autostereoscopy.Display devices 124 a-124 n may also be a head-mounted display (HMD). Insome embodiments, display devices 124 a-124 n or the corresponding I/Ocontrollers 123 may be controlled through or have hardware support forOPENGL or DIRECTX API or other graphics libraries.

In some embodiments, the computing device 100 may include or connect tomultiple display devices 124 a-124 n, which each may be of the same ordifferent type and/or form. As such, any of the I/O devices 130 a-130 nand/or the I/O controller 123 may include any type and/or form ofsuitable hardware, software, or combination of hardware and software tosupport, enable or provide for the connection and use of multipledisplay devices 124 a-124 n by the computing device 100. For example,the computing device 100 may include any type and/or form of videoadapter, video card, driver, and/or library to interface, communicate,connect or otherwise use the display devices 124 a-124 n. In oneembodiment, a video adapter may include multiple connectors to interfaceto multiple display devices 124 a-124 n. In other embodiments, thecomputing device 100 may include multiple video adapters, with eachvideo adapter connected to one or more of the display devices 124 a-124n. In some embodiments, any portion of the operating system of thecomputing device 100 may be configured for using multiple displays 124a-124 n. In other embodiments, one or more of the display devices 124a-124 n may be provided by one or more other computing devices 100 a or100 b connected to the computing device 100, via the network 104. Insome embodiments software may be designed and constructed to use anothercomputer's display device as a second display device 124 a for thecomputing device 100. For example, in one embodiment, an Apple iPad mayconnect to a computing device 100 and use the display of the device 100as an additional display screen that may be used as an extended desktop.One ordinarily skilled in the art will recognize and appreciate thevarious ways and embodiments that a computing device 100 may beconfigured to have multiple display devices 124 a-124 n.

Referring again to FIG. 1C, the computing device 100 may comprise astorage device 128 (e.g. one or more hard disk drives or redundantarrays of independent disks) for storing an operating system or otherrelated software, and for storing application software programs such asany program related to the software for the screenshot linking system120. Examples of storage device 128 include, e.g., hard disk drive(HDD); optical drive including CD drive, DVD drive, or BLU-RAY drive;solid-state drive (SSD); USB flash drive; or any other device suitablefor storing data. Some storage devices may include multiple volatile andnon-volatile memories, including, e.g., solid state hybrid drives thatcombine hard disks with solid state cache. Some storage device 128 maybe non-volatile, mutable, or read-only. Some storage device 128 may beinternal and connect to the computing device 100 via a bus 150. Somestorage devices 128 may be external and connect to the computing device100 via an I/O device 130 that provides an external bus. Some storagedevice 128 may connect to the computing device 100 via the networkinterface 118 over a network 104, including, e.g., the Remote Disk forMACBOOK AIR by Apple. Some client devices 100 may not require anon-volatile storage device 128 and may be thin clients or zero clients102. Some storage device 128 may also be used as an installation device116, and may be suitable for installing software and programs.Additionally, the operating system and the software can be run from abootable medium, for example, a bootable CD, e.g. KNOPPIX, a bootable CDfor GNU/Linux that is available as a GNU/Linux distribution fromknoppix.net.

Client device 100 may also install software or application from anapplication distribution platform. Examples of application distributionplatforms include the App Store for iOS provided by Apple, Inc., the MacApp Store provided by Apple, Inc., GOOGLE PLAY for Android OS providedby Google Inc., Chrome Webstore for CHROME OS provided by Google Inc.,and Amazon Appstore for Android OS and KINDLE FIRE provided byAmazon.com, Inc. An application distribution platform may facilitateinstallation of software on a client device 102. An applicationdistribution platform may include a repository of applications on aserver 106 or a cloud 108, which the clients 102 a-102 n may access overa network 104. An application distribution platform may includeapplication developed and provided by various developers. A user of aclient device 102 may select, purchase and/or download an applicationvia the application distribution platform.

Furthermore, the computing device 100 may include a network interface118 to interface to the network 104 through a variety of connectionsincluding, but not limited to, standard telephone lines LAN or WAN links(e.g., 802.11, T1, T3, Gigabit Ethernet, Infiniband), broadbandconnections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet,Ethernet-over-SONET, ADSL, VDSL, BPON, GPON, fiber optical includingFiOS), wireless connections, or some combination of any or all of theabove. Connections can be established using a variety of communicationprotocols (e.g., TCP/IP, Ethernet, ARCNET, SONET, SDH, Fiber DistributedData Interface (FDDI), IEEE 802.11a/b/g/n/ac CDMA, GSM, WiMax and directasynchronous connections). In one embodiment, the computing device 100communicates with other computing devices 100′ via any type and/or formof gateway or tunneling protocol e.g. Secure Socket Layer (SSL) orTransport Layer Security (TLS), or the Citrix Gateway Protocolmanufactured by Citrix Systems, Inc. of Ft. Lauderdale, Fla. The networkinterface 118 may comprise a built-in network adapter, network interfacecard, PCMCIA network card, EXPRESSCARD network card, card bus networkadapter, wireless network adapter, USB network adapter, modem or anyother device suitable for interfacing the computing device 100 to anytype of network capable of communication and performing the operationsdescribed herein.

A computing device 100 of the sort depicted in FIGS. 1B and 1C mayoperate under the control of an operating system, which controlsscheduling of tasks and access to system resources. The computing device100 can be running any operating system such as any of the versions ofthe MICROSOFT WINDOWS operating systems, the different releases of theUnix and Linux operating systems, any version of the MAC OS forMacintosh computers, any embedded operating system, any real-timeoperating system, any open source operating system, any proprietaryoperating system, any operating systems for mobile computing devices, orany other operating system capable of running on the computing deviceand performing the operations described herein. Typical operatingsystems include, but are not limited to: WINDOWS 2000, WINDOWS Server2022, WINDOWS CE, WINDOWS Phone, WINDOWS XP, WINDOWS VISTA, and WINDOWS7, WINDOWS RT, and WINDOWS 8 all of which are manufactured by MicrosoftCorporation of Redmond, Wash.; MAC OS and iOS, manufactured by Apple,Inc. of Cupertino, Calif.; and Linux, a freely-available operatingsystem, e.g. Linux Mint distribution (“distro”) or Ubuntu, distributedby Canonical Ltd. of London, United Kingdom; or Unix or other Unix-likederivative operating systems; and Android, designed by Google, ofMountain View, Calif., among others. Some operating systems, including,e.g., the CHROME OS by Google, may be used on zero clients or thinclients, including, e.g., CHROMEBOOKS.

The computer system 100 can be any workstation, telephone, desktopcomputer, laptop or notebook computer, netbook, ULTRABOOK, tablet,server, handheld computer, mobile telephone, smartphone or otherportable telecommunications device, media playing device, a gamingsystem, mobile computing device, or any other type and/or form ofcomputing, telecommunications or media device that is capable ofcommunication. The computer system 100 has sufficient processor powerand memory capacity to perform the operations described herein. In someembodiments, the computing device 100 may have different processors,operating systems, and input devices consistent with the device. TheSamsung GALAXY smartphones, e.g., operate under the control of Androidoperating system developed by Google, Inc. GALAXY smartphones receiveinput via a touch interface.

In some embodiments, the computing device 100 is a gaming system. Forexample, the computer system 100 may comprise a PLAYSTATION 3, orPERSONAL PLAYSTATION PORTABLE (PSP), or a PLAYSTATION VITA devicemanufactured by the Sony Corporation of Tokyo, Japan, a NINTENDO DS,NINTENDO 3DS, NINTENDO WII, or a NINTENDO WII U device manufactured byNintendo Co., Ltd., of Kyoto, Japan, an XBOX 360 device manufactured bythe Microsoft Corporation of Redmond, Wash.

In some embodiments, the computing device 100 is a digital audio playersuch as the Apple IPOD, IPOD Touch, and IPOD NANO lines of devices,manufactured by Apple Computer of Cupertino, Calif. Some digital audioplayers may have other functionality, including, e.g., a gaming systemor any functionality made available by an application from a digitalapplication distribution platform. For example, the IPOD Touch mayaccess the Apple App Store. In some embodiments, the computing device100 is a portable media player or digital audio player supporting fileformats including, but not limited to, MP3, WAV, M4A/AAC, WMA ProtectedAAC, AIFF, Audible audiobook, Apple Lossless audio file formats and.mov, .m4v, and .mp4 MPEG-4 (H.264/MPEG-4 AVC) video file formats.

In some embodiments, the computing device 100 is a tablet e.g. the IPADline of devices by Apple; GALAXY TAB family of devices by Samsung; orKINDLE FIRE, by Amazon.com, Inc. of Seattle, Wash. In other embodiments,the computing device 100 is an eBook reader, e.g. the KINDLE family ofdevices by Amazon.com, or NOOK family of devices by Barnes & Noble, Inc.of New York City, N.Y.

In some embodiments, the communications device 102 includes acombination of devices, e.g. a smartphone combined with a digital audioplayer or portable media player. For example, one of these embodimentsis a smartphone, e.g. the IPHONE family of smartphones manufactured byApple, Inc.; a Samsung GALAXY family of smartphones manufactured bySamsung, Inc.; or a Motorola DROID family of smartphones. In yet anotherembodiment, the communications device 102 is a laptop or desktopcomputer equipped with a web browser and a microphone and speakersystem, e.g. a telephony headset. In these embodiments, thecommunications devices 102 are web-enabled and can receive and initiatephone calls. In some embodiments, a laptop or desktop computer is alsoequipped with a webcam or other video capture device that enables videochat and video call.

In some embodiments, the status of one or more machines 102, 106 in thenetwork 104 are monitored, generally as part of network management. Inone of these embodiments, the status of a machine may include anidentification of load information (e.g., the number of processes on themachine, CPU and memory utilization), of port information (e.g., thenumber of available communication ports and the port addresses), or ofsession status (e.g., the duration and type of processes, and whether aprocess is active or idle). In another of these embodiments, thisinformation may be identified by a plurality of metrics, and theplurality of metrics can be applied at least in part towards decisionsin load distribution, network traffic management, and network failurerecovery as well as any aspects of operations of the present solutiondescribed herein. Aspects of the operating environments and componentsdescribed above will become apparent in the context of the systems andmethods disclosed herein.

B. Linking a Screenshot to Other Content

Systems and methods of the present solution are directed to linkingscreenshots to other resources. Consumer products companies oftenproduce content that is published via the Internet. For example, acompany may have its own website including web pages dedicated to itsproducts or to brands associated with the company. In some instances,companies may promote their brands and products through third party webpages, such as social media sites. For example, many companies have oneor more accounts on websites such as Facebook, Instagram, Pinterest, andTwitter. Users can view content published by a company by visiting thecompany's web pages or third party web pages on which the companypublishes content. In some cases, users also may view content publishedby a company using a dedicated application on a client device, such asan application associated with a social media website on which thecompany publishes content.

Users can save to their devices images of content that interests them onthe Internet. Users can accomplish this by capturing a screenshot of adisplay of the client device while the client device displays selectedcontent. Typically, a screenshot refers to an image representing aportion or the entire graphical user interface of a display of a clientdevice. Thus, a screenshot captured while the client device is beingused to display content published by a company (for example, in a webbrowser or a third party application), will include the relevant contentthat the user wishes to save, and may also include other elements of thegraphical user interface of the client device. A screenshot is oftensaved as an image file on the client device. In some client devices, thescreenshots are saved in an image folder of the client device. The imagefolder is capable of storing the screenshots captured by the clientdevice, regardless of the content or the application displayed on theclient device at the time the screenshot was captured. In this way, theimage folder can store screenshots of content displayed by multipledifferent applications of the client device.

Because screenshots typically may primarily include image data, it canbe difficult for a user to access content relevant to what is depictedin a screenshot when the user only has access to the screenshot itself.For example, while the screenshot may provide an image of a product thatthe user would like to purchase, the image file associated with thescreenshot itself does not provide any interface with which the user caninteract to purchase the product shown in the screenshot. The presentdisclosure provides techniques for linking a screenshot to otherrelevant resource and/or content, such as a shopping interface.

One aspect of the present disclosure can be implemented in a method orsystem for providing an online store associated with a screenshot. Ascreenshot linking system including one or more processors can identifya screenshot from images stored on a device, such as a photo roll on amobile device storing both photos and screenshots. The screenshotlinking system can identify an image included within the screenshot. Thescreenshot linking system can match the identified image with areference image of a plurality of reference images maintained by thescreenshot linking system. Each reference image can be associated with alink to content. In some implementations, the link can be a link tocontent on a mobile application or content to a webpage. In someimplementations, the content to which the reference image is linked caninclude the reference image. Responsive to the screenshot linking systemmatching the image with a reference image, the screenshot linking systemcan then identify a link associated with the image. In someimplementations, the screenshot linking system can generate and providea notification to the device associated with the screenshot. Thenotification can include the link to content corresponding to the imageidentified in the screenshot. In some implementations, if the image inthe screenshot is an image of a product, the link provided in thenotification can be a link to a page of a mobile application or a webserver through which the user of the device can purchase or otherwisereceive additional information about the product. In someimplementations, responsive to the user visiting the page via the link,the screenshot linking system can receive, from the device, a request topurchase the product and execute the request to purchase the product. Insome implementations, it can be determined that an image filecorresponds to a screenshot, and the screenshot can be analyzed toidentify at least one application associated with the screenshot.

Referring now to FIG. 2A, a block diagram depicting an embodiment of asystem 201 for linking a screenshot to other resources or content isshown. In brief overview, the system 201 can include a client device,such as the client 102, and a server, such as the server 106. The clientdevice can include a plurality of applications 204 a-204 n, an imagedatabase 210, and a screenshot linking application 215. The server caninclude a screenshot linking system 203 having a user account manager224, a publisher account manager 226, a link generator 227, a shopinterface generator 229, a notification generator 230, a contentrecommendation provider 231, and a database 232.

The client 102 can be any type of computing device with which a user caninteract directly. For example, in some implementations, the client 102can be a desktop computer, a laptop computer, a tablet computing device,a smartphone, or a virtual reality computing device. The applications204 can be installed on the client 102 to allow the user to accessvarious forms of content. For example, one or more of the applications204 may correspond to web browsers configured to display web pages.Other applications 204 may correspond to social media applications, suchas Facebook, Instagram, Pinterest, and Twitter. Still other applications204 may be applications provided by consumer products companies thatallow a user of the client 102 to view information about products madeavailable by the companies who develop the applications 204.Applications 204 also can correspond to various other utilities such ascalendars, clocks, word processing applications, mapping applications,news providers, messaging applications, email applications, and thelike. Computer instructions corresponding to each of the applications204 may be stored in a memory associated with the client 102. When auser executes a particular application 204, graphical output from theapplication 204 can be displayed on a display screen associated with theclient 102.

In some implementations, the client 102 can execute an operating systemthat provides a file directory structure and various user interfaceelements. For example, the operating system of the client 102 can beconfigured to allow a user to capture a screenshot of a graphical userinterface of the client device 102. In some implementations, ascreenshot may be an image representing all of the graphical outputdisplayed on a display screen of the client 102 at the time thescreenshot is captured. A screenshot can be an image file depicting thisgraphical output. In some implementations, screenshots may be stored onthe client device in the image database 210. For example, the imagedatabase 210 can correspond to a file directory provided by theoperating system of the client 102. In some implementations, the imagedatabase 210 can be a portion of a larger memory associated with theclient 102 that is dedicated to storing images, including screenshotsthat may be captured using a screenshot capture function of the client102. In some implementations, when a user captures a screenshot using animage capture function of the client 102, the resulting image file canbe saved automatically in the image database 210. In some otherimplementations, a user may manually select the file directorycorresponding to the image database for storing a screenshot, after thescreenshot has been captured on the client 102.

The file directory corresponding to the image database 210 can be sharedamong (e.g., accessible by) multiple applications 204. In someimplementations, various applications 204 of the client 102 can accessthe image database 210. For example, screenshots captured when any ofthe various applications 204 is displayed on the device may beautomatically saved in the image database 210. The image database 210can therefore serve as a shared memory between the various applications204 on the client 102. As a result, the screenshots saved from manydifferent applications can be accessed in a single location, such as theimage database 210. In some implementations, the client 102 also mayinclude a screenshot receiver, which may be implemented ascomputer-executable instructions that may execute, for example, as partof an operating system executing on the client 102. Any of the variousapplications 204 may use the screenshot receiver to access screenshotsstored in the image database 210. In contrast, most other data collectedfrom an application 204 is stored in a separate folder or portion ofmemory that is separated from the data stored by other applications 204.

The client device 102 also includes a screenshot linking application215. The screenshot linking application can be executed on the clientdevice in a manner similar to the applications 204 discussed above. Thescreenshot linking application 215 can include a screenshot analyzer 220and an image analyzer 225. In some implementations, the screenshotlinking application 215 can interact with the image database 210 toretrieve and process image files stored in the image database 210. Forexample, the screenshot linking application 215 can retrieve an imagefile corresponding to a screenshot from the image database 210. Thescreenshot analyzer 220 of the screenshot linking application 215 canprocess the screenshot image file, for example to extract one or moreimages from the screenshot. The image analyzer can then process theextracted images, as discussed further below. In some implementations,the screenshot linking application 215 can be configured to retrievemultiple image files corresponding to screenshots from the imagedatabase 210. In some implementations, the screenshot linkingapplication 215 can identify the storage location, for example, theimage database 210, where screenshots captured by the device are storedand then identify multiple image files corresponding to the screenshotscaptured by the device. In some implementations, the screenshot linkingapplication can identify screenshots from other types of images filesusing image processing techniques, analyzing metadata associated withthe screenshot files, among others.

The screenshot linking system 203 executing on the server 106 includes auser account manager 224 and a publisher account manager 226. In someimplementations, the user account manager 224 can be used to create anaccount for a user of the client device 102, for example by collecting,storing, and modifying information relating to a user of the clientdevice 102. For example, the user account manager 224 can storeinformation such as a name, gender, age, email address, telephonenumber, and demographic information associated with a user of the clientdevice 102. In some implementations, the user can input such informationto the user account manager via a user interface provided on the clientdevice 102 by the screenshot linking application 215. The user accountmanager 224 also can store information relating to the client 102, suchas an IP address, a MAC address, a model number, or a serial number. Allof the information associated with the user account can be stored in thedatabase 232. In some implementations, information corresponding to theuser account can be used to associate communications received from aclient 102 with a particular user. For example, the user account manager224 can determine a source address of communications originating from aclient 102 and can compare the source address to information associatedwith user accounts stored in the database 232 to identify which clientdevice 102 is associated with a particular communication.

The publisher account manager 226 can be used to create an account for acontent publisher, such as a business. For example, the publisheraccount manager 226 can create an account including identificationinformation associated with a content publisher by collecting suchinformation from the content publisher and storing the information inthe database 232. In some implementations, the publisher account manager226 also can store information relating to content provided by thecontent publisher. For example, the publisher account manager 226 canreceive images, such as those used in advertisements published by thecontent publisher. In some implementations, such content can be storedin the database 232 and associated with the account of the contentpublisher.

The screenshot linking system 203 also includes the link generator 227,which can be used to generate a link associating image data receivedfrom the client 102 with another resource, such as a URL or anapplication. For example, in some implementations the link generator 227can map an image to a resource. The shop interface generator 229 can beused to generate an interface through which a user of the client 102 canpurchase products. In some implementations, the interface generated bythe shop interface generator 229 can be provided to the client 102,allowing a user to make purchases via the client 102. The notificationgenerator 230 can generate information corresponding to a notificationand can transmit the notification to the client 102 to alert the user ofthe client 102 that the notification has been generated. In someimplementations, a notification can include a link generated by the linkgenerator 227. The content recommendation provider 231 can receiveinformation relating to activity of the user of the client 102, and cangenerate a recommendation of a product that may be of use to the user.The recommendation can then be provided to the client 102, for examplein the form of a notification. These components are discussed furtherbelow.

It should be understood that the block diagram of FIG. 2A is intended tobe illustrative only. For example, while components of the system 201are shown as executing on either the client 102 or the server 106, insome implementations, components shown as executing on the client 102may instead be executed on the server 106. Similarly, components shownas executing on the server 106 may instead be executed on the client102. In some implementations, one instance of a component, such as thescreenshot analyzer 220, may be executed on the client 102 while asecond instance of the same component is executing on the server 106.

Referring now to FIG. 2B, a conceptual diagram depicting an example usecase for the system 201 of FIG. 2A is shown. The client 102 isconfigured to send image data 236 to the server 106. The server 106 isalso configured to transmit information to the client 102. As shown inFIG. 2B, the image database 210 includes a screenshot 238, whichincludes an image 240. In some implementations, the screenshot linkingapplication 215 can receive the screenshot 238 from the image database210. For example, the screenshot linking application 215 maycontinuously or periodically monitor the image database 210 for newscreenshots 238, and may retrieve the screenshot 238 from the imagedatabase 210. For example, in some implementations, the screenshotlinking application 215 can scan the image database 210 and sequentiallyprocess each image stored in the image database 210 according to thetechniques described further below. In some implementations, thescreenshot linking application 215 itself may process the screenshot 238to extract the image 240. For example, the screenshot linkingapplication may include an image extractor configured to extract theimage 240 from the screenshot 238. In some other implementations, thescreenshot linking application 215 may send image data 236 correspondingto the screenshot 238 to the screenshot linking system 203 on the server106, where the screenshot linking system 203 may process the screenshot238 to extract the image 240. After the image 240 has been extracted, alink associated with the image can be identified. The link itself maynot be included in the screenshot 238. For example, in someimplementations, the image 240 can be matched to a reference image thatcorresponds to an image associated with the account of a contentprovider, as discussed above. The reference image can be associated witha link, and the link can be associated with the screenshot 238responsive to determining the match. The match can be determined locallyon the client 102, or remotely by the server 106. In some otherimplementations, the link may be determined without comparing the image240 to a reference image. For example, a watermark or fingerprintassociated with the image 240 can be used to determine the link, asdiscussed further below. In some implementations, reference images canbe retrieved from social media websites on which content publisherspromote products. For example, the posts of a particular company can bescanned on a social media site and fingerprints can be generated foreach image. The image and the generated fingerprints can then be savedand can then serve as reference images.

In some implementations, the screenshot linking system 203 can determinethe number of times that a particular piece of content provided by acontent publisher has been captured in a screenshot by clients such asthe client 102. For example, for each published content item (e.g.,images, social media posts, etc.) provided by a content provider, thescreenshot linking system 203 can maintain a counter that is increasedeach time the screenshot linking system 203 receives image data 236indicating that a screenshot including the content item has beencaptured. The counter can be increased in response to receiving suchimage data from any number of clients similar to the client 102. Thus,the counter for each content item can serve as a measure of popularityfor each content item. In some implementations, the counter informationcan be provided to a user of the client 102. For example, when ascreenshot is captured on the client 102, the screenshot linking system203 can send to the client 102 information representing the number oftimes other users of other client devices have captured screenshots withthe same content item featured in the screenshot captured by the client102. In some implementations, the counter information for each contentitem also can be provided to the content publishers. For example,content publishers can use this information to better understand whichcontent items are likely to be captured in screenshots by users ofclient devices, such as the client 102. In some implementations, acontent publisher can be notified each time a user captures a screenshotfeaturing a content item that was provided by the content publisher. Acontent publisher also can be provided with information regarding theidentity of users who have captured screenshots featuring content itemspublished by the content publisher. In some implementations, thescreenshot linking application 215 can provide an interface throughwhich a content publisher can communicate with a user of the client 102,in response to a determination that the client 102 has captured ascreenshot featuring a content item provided by the content publisher.

A notification 242 including the link 246 can then be generated. In someimplementations, the notification 242 can be generated on the client102. In some other implementations, the notification 242 can begenerated by the screenshot linking system 203 executing on the server106. In some implementations, reference images can be retrieved fromsocial media websites on which content publishers promote products. Forexample, the posts of a particular company can be scanned on a socialmedia site and fingerprints can be generated for each image. The imageand the generated fingerprints can then be saved and can then serve asreference images. The notification 242 can alert a user of the client102 that a link 246 associated with the screenshot 240 has beenidentified. In some implementations, the link 246 can be or can includea URL. For example, the URL can direct a web browser executing on theclient 102 to a website operated by the company who published the image240, responsive to a user of the client 102 selecting the link 246. Thewebsite can be configured to display information relating to a productshown in the image 240 of the screenshot 238.

In some implementations, the notification 242 can be displayed on theclient 102 over the active application 204. For example, if a screenshotis taken while a particular application 204 is active on the client 102,the screenshot may be processed as discussed above and the notification242 can be displayed over a view port of an active application 204currently displayed on the device. In some implementations, thenotification 242 can be a popup notification. The notification 242 mayinclude text notifying a user of the client 102 that a link 246associated with the most recently captured screenshot has beenidentified. In some implementations, the link 246 may be displayed inthe notification 242. In some other implementations, a user can selectthe notification 242 to be directed to a different page or application204 in which the associated link 246 is displayed. In someimplementations, the notification can be a local notification generatedby the screenshot linking application executing on the device. In someimplementations, the notification can be a push or remote notificationthat is triggered by a server of the screenshot linking system andreceived by the device, upon which the notification is displayed.

In some other implementations, the link 246 can link to a particularpost that has been viewed on one of the applications 204. For example,selecting the link 246 can cause one of the applications 204 to open andto display a post containing an image matching the image 240 that isincluded in the screenshot 238. Thus, a user may select the link 246 tobe directed to the original post that was displayed on the client 102when the screenshot 238 was captured.

In still other implementations, the link 246 can direct a user to ageneric shopping interface that may not be associated with the companywho published the image 240. For example, the shopping interface may beprovided by the operator of the screenshot linking system 203, or by athird party. The shopping interface can allow a user to purchase aproduct shown in the image 240, as discussed further below.

FIGS. 2C-2G illustrate various stages of a process for linking ascreenshot to a product purchase page. FIG. 2C shows a screenshotcaptured by a user of the client 102. The screenshot includes an image274 including a product (in this case, a shirt), as well as various uservisual features, such as interface elements. As described above, anddiscussed further below, the screenshot shown in FIG. 2C can beprocessed to extract the image 274, and to identify a link associatedwith the image 274. A notification 276 including the link can then beprovided, as shown in FIG. 2D. A user can select the link includedwithin the notification 276, which can cause the client 102 to displaythe shopping interface shown in FIG. 2E. As shown, the shoppinginterface can include a picture of the original image 274, as well as abutton 278 that can be selected if a user wishes to purchase the productshown in the image 274. The shopping interface also can includeadditional information related to the product, such as a description 280and the name of the company who produces the product (in this instance,Staple Pigeon). Selecting the button 278 can cause the shoppinginterface to display additional selectable options 282 related to theproduct, such as the desired size and color of the product, as shown inFIG. 2F. An “add to cart” button 284 can be selected after the user hasselected one or more of the options 282. After the product is added tothe cart, the shopping interface can display a checkout screen,including a checkout button 286. Selecting the checkout button 286 cancomplete the sale of the product to the user.

In some implementations, the user account manager 224 can store paymentinformation, such as a credit card number and shipping address, in thedatabase 232. Selecting the checkout button 286 can then cause theproduct to be billed to the credit card associated with the user accountof the client 102, and can notify a retailer that the selected productshould be sent to the shipping address associated with the user accountof the client 102. In some other implementations, the shopping interfacemay prompt the user to enter payment information and a shipping addressmanually in response to the user selecting the checkout button 286.

FIG. 2H is a flow chart of a process 250 for linking a screenshot toother content. In brief overview, it can be determined that an imagefile corresponds to a screenshot (step 252), the screenshot can beanalyzed to identify at least one application associated with thescreenshot (step 254), and a link can be generated associating thescreenshot with a resource (step 256).

Referring again to FIG. 2H, and in greater detail, the process 250includes determining that an image file corresponds to a screenshot(step 252). Such a determination may be made, for example, by thescreenshot analyzer 220 shown in FIG. 2A, which may include a screenshotdetector, as described further below in connection with FIG. 3A. In someimplementations, an image file can be obtained, retrieved or scannedfrom the image database 210. Characteristics of the image file can thenbe analyzed to determine whether the image file corresponds to ascreenshot. For example, in some implementations, the resolution of theimage file can be compared to the resolution of a display screen of theclient 102. Because a screenshot captures an image representing what isdisplayed on a display screen of the client 102, a screenshot typicallywill have approximately the same resolution as the display screen of theclient 102. Thus, in some implementations, the resolution of the displayscreen of the client 102 can be determined to compare the resolution ofimages on the client 102 to the resolution of the display screen. Forexample, the client 102 can have a display screen with a standard highdefinition (HD) resolution of 1920-by-1080, which is typical among manysmartphones available. The screenshot linking application can determinea resolution of an image file, and if the resolution of the image filematches or is close to the resolution of the display screen of theclient 102, the screenshot linking application can determine that theimage file is a screenshot. Other resolutions are also possible for thedisplay screen of the client 102. In some implementations, files havingrelatively high resolutions can be more likely to have been captured bya camera than captured as a screenshot. For instance, the resolution ofa screenshot taken on an APPLE IPHONE 6 is 750×1334 (which is based onthe resolution of the display screen), while the resolution of an imagetaken by the rear camera of the APPLE IPHONE 6 is 8 MP (which is basedon the resolution of the camera). Other mobile devices may captureimages and screenshots having different resolutions. In some otherimplementations, metadata associated with the image file can be used todetermine whether the image file corresponds to a screenshot. Forexample, metadata may include an indication of an origin of the imagefile, which can help to determine whether the image file represents ascreenshot. If metadata associated with the image file indicates thatthe image file was received by the client 102 from a remote computingdevice, it may be unlikely that the image file represents a screenshotcaptured on the client 102. In other implementations, othercharacteristics of the image file can be analyzed to determine whetherthe image file represents a screenshot. For example, in someimplementations, the determination of whether the image file representsa screenshot can be based on a file size of the image file. In someimplementations, smaller files may be more likely to have been capturedas screenshots than as photos, for example.

After it has been determined that the image file corresponds to ascreenshot, the screenshot can be analyzed to identify at least oneapplication associated with the screenshot (step 254). Thisidentification can be made, for example, by the screenshot analyzer 220shown in FIG. 2A, which may include an application detector, asdescribed further below. In some implementations, visual elements of theimage file can be correlated with certain applications. In otherimplementations, metadata of the image file can be used to determine anapplication associated with the screenshot. Additional details regardingtechniques for identifying an application associated with a screenshotare discussed below in Section B.

A link can then be generated associating the screenshot with a resource(step 256). In some implementations, the link can be generated by thelink generator 227 shown in FIG. 2A. The generated link can be based ona product shown in an image within the screenshot. For example, thescreenshot can include an image showing a particular product, and thelink can be a URL associated with a web page that allows a user toreceive more information about the product or to purchase the product.In some other implementations, the link can be directed to a particularpost that has been viewed on one of the applications that executes onthe client 102. In still other implementations, the link can direct auser to a generic shopping interface, such as the shopping interfaceshown in FIGS. 2C-2G. In each of these examples, a user may select thelink from the notification, for example using a pointing device or otherinput device to select the link. Selection of the link may cause aresource display component of the client 102 to access the resourceassociated with the link (i.e., to display graphical content associatedwith the link).

FIG. 2I is a flow chart of a process 260 for providing an online storeassociated with a screenshot. In brief overview, a screenshot can beidentified (step 262). An image included within the screenshot can beidentified (step 264). A link associated with the image can beidentified (step 266). A notification including the link can be provided(step 268). A page can be provided via the link to purchase a productdisplayed in the image (step 270). A request to purchase the product canthen be received and executed (step 272).

Referring again to FIG. 2I, and in greater detail, the process 260 caninclude identifying a screenshot (step 262). As discussed above, imagefiles can be retrieved from the image database 210. In someimplementations, the screenshot linking application 215 can monitor theimage database 210 to determine when image files are added, and canprocess image files as they are added to the database 210 or otherwiseon a predetermined frequency. In some implementations, image files canbe processed to determine whether the image files correspond toscreenshots, for example by examining dimensions of the image files suchas file size and/or resolution. In some implementations, screenshots canbe identified by the screenshot analyzer 220 shown in FIG. 2A.

The process 260 can include identifying an image included within thescreenshot (step 264). In some implementations, the screenshot can beprocessed to identify and/or extract an image. Various techniques can beemployed to identify and/or extract the image from the screenshot, asdiscussed further below in Section D. In some implementations, acomponent of the screenshot linking application 215 executing on theclient 102, such as the screenshot analyzer 220 or the image analyzer225 shown in FIG. 2A can extract the image from the screenshot. In otherimplementations, the screenshot can be transmitted from the client 102to the server 106, and the image can be extracted from the screenshot bya component of the screenshot linking system 203 executing on the server106.

The process 260 includes identifying a link associated with the image(step 266). In some implementations, the link can be generated by thelink generator 227 shown in FIG. 2A and may be identified by a linkidentifier executing on the client 102. The link can be based on aproduct shown in the image extracted from screenshot. For example, thelink can be a URL associated with a web page that allows a user toreceive more information about a product shown in the image, or topurchase the product. In some other implementations, the link can bedirected to a particular post that has been viewed on one of theapplications that executes on the client 102.

In some implementations, the link can be a deep link. For example, adeep link can refer to a link to a particular page or portion of aparticular application, such as the application that was used togenerate the screenshot with which the link is associated. Thus, thelink can direct a user to a certain portion or page of an application,rather than merely launching the application. In some implementations, adeep link can be a URL link to a particular piece of content on awebsite, for example a particular page, instead of the homepage of thewebsite. When the user selects the link, the user can be broughtdirectly to the relevant content on the website.

As discussed above, such a link can be referred to as a “deep link.” Instill other implementations, the link can direct a user to a genericshopping interface, such as the shopping interface shown in FIGS. 2C-2G.

The process 260 includes providing a notification including the link(step 268). In some implementations, the notification can be generatedon the client 102, for example by the screenshot linking application215. In some other implementations, the notification can be generated bythe screenshot linking system 203 executing on the server 106 andtransmitted to the client 102. The notification can alert a user of theclient 102 that a link associated with the screenshot has beenidentified. The user of the client 102 can then select the link fromwithin the notification to be directed to the resource associated withthe link.

The process 260 includes providing, via the link, a page to purchase aproduct displayed in the image (step 270). In some implementations, thepage may be provided by a web browser application executing on acomputing device such as the client 102 shown in FIG. 2A. For example,as discussed above, the link may direct a user to a website operated bythe company who produces the product. The user may then use thecompany's website to complete the purchase. In other implementations,the link can direct a user to a generic shopping interface or universalstore that may not be associated with the company who produces theproduct. For example, the shopping interface may be provided by theoperator of the screenshot linking system, or by a third party. Theuniversal store can include an interface that allows the user to selectvarious options related to the product, such as the desired size andcolor of the product. The interface for the universal store can bepresented directly over other content that may be displayed on theclient device, such as the image that is associated with the link. Forexample, the universal store interface can be overlaid with the imageassociated with the link, allowing the user to purchase a product shownin the image after selecting various options related to the product.

In some implementations, the universal store can include or cancommunicate with a database containing information relating to theproducts offered for sale through the universal store. For example, thescreenshot linking application can communicate with the database toretrieve information about the product associated with a link that theuser selected. The database can include information such as themanufacturer, the item name and serial number, the price of the item,and various selectable options associated with the item. The universalstore interface can be overlaid on an image associated with the product,and can display information about the product obtained from thedatabase. In some implementations, the universal store can include userinterface elements that allow a user to select various optionsassociated with the item, such as size and color.

The universal store can be independent of the web site associated withthe manufacturer of the product that is offered for sale through theuniversal store. However, in some implementations, the user'sinteractions with the interface of the universal store may later be sentto a web site of the manufacturer of the product that is offered forsale through the universal store. For example, the user may purchase theproduct through the universal store interface, and information relatedto the sale, such as the size and color of the product, as well asshipping and billing information, can be sent to the manufacturer of theproduct that is offered for sale through the universal store. Theprocess 260 includes receiving and executing a request to purchase theproduct (step 272). In some implementations, the request can be receivedby the screenshot linking application 215 executing on the client device102. In some implementations, the page can include a checkout buttonthat the user can select to complete the sale of the product to theuser. For example, payment information, such as a credit card number andshipping address, can be stored in a user account of the user of theclient 102. Selecting the checkout button can then cause the product tobe billed to the credit card associated with the user account of theclient 102, and can notify a retailer that the selected product shouldbe sent to the shipping address associated with the user account of theclient 102. In some other implementations, the shopping interface mayprompt the user to enter payment information and a shipping addressmanually in response to the user selecting the checkout button.

C. Determining an Application Associated with a Screenshot

As discussed above, users of computing devices may wish to save imagesof content that interests them, such as from web pages on the Internetor screens of mobile applications, in the form of screenshots. Ascreenshot can be processed to generate a link to other relevantcontent. In some implementations, it may be useful to determine anapplication associated with a screenshot for purposes of generating sucha link. For example, a deep link may be a link to a particular page orportion of a particular application, such as the application that wasused to generate the screenshot. Therefore, for the screenshot linkingsystem to generate a deep link, the screenshot linking system mayidentify the identity of the application active on the display of theclient device at the time the screenshot was captured. Furthermore, thescreenshot linking system may also be useful to process only a subset ofthe screenshots saved on a client device, such as screenshots that aregenerated by only a subset of applications. For example, someapplications may be added to a white list of applications whosescreenshots are to be processed and linked to other content. Screenshotsassociated with applications not included in the white list may not beprocessed in order to save computing resources.

One aspect of this disclosure can be implemented in a method or systemfor determining an identity of an application that is associated with aparticular screenshot. The screenshot linking system can identify animage file and determine that the image file corresponds to ascreenshot. The screenshot linking system can identify one or morevisual elements of the screenshot. The screenshot linking system canthen determine an identity of an application corresponding to thescreenshot based on the identified visual elements.

Referring now to FIG. 3A, a block diagram depicting an embodiment of ascreenshot analyzer 220 that can be used in the system of FIG. 2A isshown. The screenshot analyzer 220 is shown as a component of thescreenshot linking application 215, which executes on the client 102.However, in other implementations, the screenshot analyzer 220 caninstead execute on the server 106. In brief overview, the screenshotanalyzer 220 includes a screenshot detector 325, an application detector330, and a policy manager 335.

The screenshot detector 325 can be configured to determine when ascreenshot is captured on the client device 102. As discussed above, insome implementations, screenshots can be saved to an image database suchas the image database 210 shown in FIG. 2A. The screenshot detector 325can continuously or periodically monitor the image database 210 todetermine when new image files are added to the image database. In someimplementations, the screenshot detector 325 can be further configuredto determine whether a newly added image file corresponds to ascreenshot or a different type of image file, for example based on theresolution of the newly added image file or metadata associated with thenewly added image file. In some implementations, the screenshot detector325 can determine if the image corresponds to a screenshot if theresolution of the screenshot is less than a predetermined resolution.

The application detector 330 can be configured to determine anapplication associated with a screenshot. In some implementations, theapplication detector 330 can be configured to determine one or moreapplications that were active on the client 102 when a particularscreenshot was captured. In some other implementations, the applicationdetector 330 can be configured to determine which application was usedto display an image that is present within a screenshot. Thedetermination can be made, for example, based on visual elements of thescreenshot. In other implementations, the determination can be madebased on other information, such as metadata associated with thescreenshot.

In some implementations, the application detector 330 may includetemplates or image files of one or more applications. The applicationdetector 330 can identify characteristics of the images of each of theapplications to generate a list of characteristics unique to theparticular application. For instance, each application may use a uniquecolor scheme, font, layout, among others. In some implementations, thelayout of the selectable icons may be unique in each of theapplications. The application detector 330 can identify text in imagesby employing optical character recognition (OCR) techniques or othertext recognition techniques. The application detector can then determinethe application associated with the screenshot based on the identifiedtext.

The application detector 330 can analyze or parse a screenshot andidentify one or more characteristics of the image, including but notlimited to the color scheme, layout, and arrangement of selectableicons, among others. The application detector 330 can then compare theidentified characteristics with known characteristics unique to each ofthe one or more applications known to the application detector 330. Theapplication detector 330 can then determine, based on the comparison ofthe characteristics of the screenshot, that the screenshot correspondsto one of the applications.

The policy manager 335 can store one or more policies related toprocessing of screenshots. For example, policies can include informationrelated to processing of screenshots based on the applications that areassociated with the screenshots. Each policy can include one or morerules. For example, the policy manager 335 can maintain a white list ofapplications. After the application detector 330 has determined anapplication associated with a screenshot, the application can becompared to the applications included in the white list maintained bythe policy manager 335. In some implementations, if the applicationdetector 330 determines that the application matches one of theapplications included in the white list, the screenshot may be furtherprocessed to identify or generate a link associating the screenshot withanother resource. However, if the application does not match one of theapplications included in the white list, the screenshot may be discardedwithout further processing.

The policy manager 335 also can implement other application-basedpolicies. For example, the policy manager 335 can assign higher priorityto screenshots associated with particular applications, such that thosescreenshots are processed more quickly than screenshots associated withlower priority applications. In some implementations, the policy manager335 can store information relating to various visual elements that areassociated with particular applications. Such information can serve asreference information for the application detector 330 when determiningwhich application a particular screenshot corresponds to.

In some implementations, the policy manager 335 can implement policiesthat control the timing of image processing or notifications. Forexample, the policy manager 335 can control the frequency with which theimage database is scanned for new image files. Thus, the image databasecan be scanned for new files once every week, once every day, once everyhour, or at any other interval according to a policy implemented by thepolicy manager 335. In some implementations, the policy manager 335 cancause an image to be processed after a predetermined interval has passedsince the most recent image has been added to the image database. Insome implementations, the policy manager 335 can cause images to beprocessed together in batches, for example after a predetermined numberof new images have been added to the image database.

In some implementations, the policy manager 335 can implement policiesto control other aspects of the screenshot linking system. For example,the policy manager 335 can control the timing with which notificationsare displayed on the client 102. In some implementations, a user of theclient can interact with the policy manager 335 to customize theparameters associated with various policies.

FIG. 3B is an example screenshot including an image 346. The presence ofthe screenshot on the client device can be detected by the screenshotdetector 325, as discussed above. After the screenshot has beendetected, the application detector 330 can analyze the screenshot todetermine an application associated with the screenshot. In someimplementations, the application detector 330 identifies and analyzesvisual elements of the screenshot to determine an application associatedwith the screenshot. Visual elements may include colors, shapes, icons,words, font styles, layouts, or any other visual features of thescreenshot. For example, the screenshot shown in FIG. 3B includes a backbutton 340 and a refresh button 342 at the top of the screenshot, anaccount name 344, an image 346, a favorite button 348, a comment button350, and various user interface elements 352.

The application detector 330 can process the screenshot to identify thevisual elements, and can identify an application associated with thescreenshot based on the identified visual elements. In someimplementations, the application detector 330 compares the visualelements identified in the screenshot to reference information stored bythe policy manager 335. For example, the policy manager 335 may storeinformation about particular colors or font styles used in a certainapplication. If the application detector 330 determines that the colorsor font styles of a screenshot match those associated with a particularapplication, the application detector 330 can determine that thescreenshot is associated with that application. This processingtechnique can be efficient because, in many cases, applicationsexecuting on the client 102 have distinctive interface designs that maycorrespond to various visual elements. In some other implementations,the application detector 330 can use metadata of a screenshot todetermine an application associated with the screenshot. For example,metadata may include text-based information indicating the applicationthat was active on the client when the screenshot was captured.

FIG. 3C is an example graphical user interface indicating an applicationassociated with a screenshot. In some implementations, the screenshotlinking application 215 can cause the graphical user interface shown inFIG. 3C to be displayed on the client 102 in response to determining anapplication associated with a particular screenshot. The graphical userinterface includes an image 354 and an application identification 356.The image 354 can correspond to an image that was present in thescreenshot that has been analyzed by the application detector 330. Insome other implementations, the graphical user interface shown in FIG.3C can include the entire screenshot, including the image 354 and othervisual elements present in the screenshot. The applicationidentification 356 includes the name of the application associated withthe screenshot (in this example, the associated application isInstagram), as well as a time stamp showing the day on which thescreenshot was captured. Thus, the graphical user interface shown inFIG. 3C can be used to remind a user of the application used to capturea particular screenshot, as well as the time at which the screenshot wascaptured.

In some implementations, the graphical user interface shown in FIG. 3Calso can include other information or other interface elements. Forexample, the graphical user interface can include a link to a profilepage associated with the content publisher who provided the image shownin the screenshot. In some implementations, the screenshot linkingapplication can scan the screenshot for text associated with the name ofthe content publisher who provided an image featured in the screenshot,for example using optical character recognition. If such text is shownin the screenshot, the graphical user interface can include a link thatis selectable by a user of the client 102. Thus, as shown in FIG. 3C,the text 355, which includes a name of a content publisher, can be aselectable link in the graphical user interface. Selecting the link candirect the user to a web page associated with the content publisher. Insome other implementations, the graphical user interface can includeother links. For example, the user interface elements 357 and 358, whichare shown in the application from which the screenshot was captured, maybe associated with links in the graphical user interface shown in FIG.3C. The links can, in some implementations, direct the user to theapplication associated with the screenshot and cause the application toperform the action associated with the user interface elements 357 and358, such as “liking” or commenting on the post from which thescreenshot was captured.

FIG. 3D is flow chart of a process 360 for determining an applicationassociated with a screenshot. In brief overview, the screenshot linkingapplication can identify an image file (step 362). The screenshotlinking application can determine that the image file corresponds to ascreenshot (step 364). The screenshot linking application can identifyone or more visual elements of the screenshot (step 366). The screenshotlinking application can determine an application corresponding to thescreenshot based on the identified visual elements (step 368).

Referring again to FIG. 3D, the process 360 can include identifying animage file (step 362). In some implementations, the screenshot linkingapplication can identify the image file. For example, the screenshotlinking application can identify image files by monitoring a filedirectory in which image files are saved, such as the image database 210shown in FIG. 2A. The screenshot linking application can then retrieveImage files from such a directory. The process 360 includes determiningthat the image file corresponds to a screenshot. In someimplementations, the determination can be made by the screenshotdetector 325, as described above. For example, the screenshot analyzer325 can monitor the image database 210 to determine when new image filesare added to the image database. In some implementations, the screenshotdetector 325 can be further configured to determine whether an imagefile corresponds to a screenshot or a different type of image file, forexample based on the resolution of the image file or metadata associatedwith the image file.

The process 360 includes identifying one or more visual elementsassociated with the screenshot (step 366). In some implementations, ascreenshot analyzer can be used to identify the one or more visualelements. Visual elements may include colors, shapes, icons, words, fontstyles, layouts, or any other visual features of the screenshot. In someimplementations, the application detector 330 may include templates orimage files of one or more applications. The application detector 330can identify characteristics of the images of each of the applicationsto generate a list of characteristics unique to the particularapplication. For instance, each application may use a unique colorscheme, font, layout, among others. In some implementations, the layoutof the selectable icons may be unique in each of the applications.

The process 360 includes determining an application corresponding to thescreenshot based on the identified visual elements (step 368). In someimplementations, an application detector can determine the applicationcorresponding to the screenshot. In some implementations, the visualelements identified in the screenshot can be compared to referenceinformation relating to visual elements associated with variousapplications. For example, the policy manager 335 may store informationabout particular colors or font styles used in a certain application. Ifit is determined that the colors or font styles of a screenshot matchthose associated with a particular application, it can also bedetermined that the screenshot is associated with that application.

D. Analyzing an Image Included within a Screenshot

As discussed above, a screenshot may capture the graphical contentdisplayed on a display screen of a client device. However, some of thevisual elements of the screenshot may be irrelevant, such as elements ofa user interface of the active application or other information that maybe displayed on a client device, such as the time, date, and batterylevel of the client device. Therefore, it can be useful to separate therelevant content of a screenshot, such as an image provided by a contentpublisher, from the less relevant content that may be included withinthe screenshot. After an image has been identified and/or extracted, theimage may be processed to determine a watermark or fingerprintassociated with the image. Watermarks and fingerprints can be used tomatch the images with reference images provided by content publishers.In this way, if a user takes a screenshot of an image that matches areference image of a content publisher, the screenshot linking systemcan provide the user, via the device of the user, a notification thatincludes a link to content corresponding to the reference image thatmatched the image of the screenshot. As a result, content publishers oradvertisers can provide relevant content to users based on screenshotscaptured by the users.

One aspect of this disclosure can be implemented in a method or systemfor analyzing an image included within a screenshot. An image can beidentified and/or extracted from a screenshot. A watermark of the imagecan be identified. The watermark can include additional informationcorresponding to the image. In some implementations, the watermark caninclude data representing a link to a page of a mobile application orweb server that includes content associated with the image. The link canbe a link to content of a product shown in the image. In someimplementations, the link can be a link to content of an advertiser orentity that published or provided the image. In some implementations,responsive to identifying the link to content, the screenshot linkingsystem can provide a notification to the client device including thelink to content to allow the user of the client device to retrieveadditional information relating to the image included in the screenshot.

Another aspect of this disclosure also can be implemented in a methodfor analyzing an image included within a screenshot. An image can beidentified and/or extracted from a screenshot. The screenshot linkingsystem or a mobile application corresponding to the screenshot linkingsystem can generate a fingerprint of the image can be generated. Thescreenshot linking system can then identify a fingerprint of a referenceimage that matches the fingerprint of the image of the screenshot. Thescreenshot linking system, responsive to determining that thefingerprint of the image of the screenshot matches the identifiedfingerprint of the reference image, can identify a link to contentassociated with the image. The screenshot linking system can thengenerate and provide a notification including the link to the user ofthe device. In this way, the screenshot linking system can provide theuser relevant content based on images included in screenshots capturedby the user. As a result, advertisers or content publishers can sendlinks relevant to the user to increase user engagement.

Referring now to FIG. 4A, a block diagram depicting an embodiment of animage analyzer 225 that can be used in the system 201 of FIG. 2A isshown. The image analyzer 225 is shown as a component of the screenshotlinking application 215, which executes on the client 102. However, inother implementations, the image analyzer 225 can instead execute on theserver 106, also shown in FIG. 2A. In brief overview, the image analyzer225 includes an entropy analyzer 430, a watermark detector 435, and afingerprint generator 440.

The entropy analyzer 430 can be used to identify and/or extract an imagefrom a screenshot. For example, as discussed above, screenshots maycontain images as well as other visual elements that are associated withthe application used to generate the screenshot. However, the user istypically more interested in the content of the image than the othervisual elements that may be included within the screenshot. Furthermore,links to other content are based on the image data, and in someimplementation should be independent of other visual elements notincluded in the image. Therefore, it can be helpful to extract an imagefrom or identify an image within a screenshot. In some implementations,the entropy analyzer 430 can determine image entropy across thescreenshot to determine which portions of the screenshot correspond toan image. In some implementations, the entropy analyzer 430 can scan thescreenshot for noise that has been introduced during the watermarkingprocess, as described further below. If such noise is present, theentropy analyzer 430 can determine that the screenshot includes awatermarked image.

Image entropy can refer to the amount of randomness associated withimage data. Thus, an image having many different values for pixel colorand pixel brightness across the image can have higher entropy than animage having relatively few different values for pixel color and pixelbrightness across the image. Often, application interfaces includevisual elements that are relatively simple in design. For example, anapplication may contain a large title bar or button that is a singlecolor, and therefore the portion of a screenshot corresponding to such avisual element can have relatively low entropy. In contrast, photosoften include pixels of many different colors and brightness levels,which can result in relatively high image entropy. The position of atitle bar or other user elements can also provide information regardingwhich application was active in the display when a particular screenshotwas captured.

Thus, in some implementations, the entropy analyzer 430 can determinethe entropy associated with various regions of a screenshot, and canpredict which regions correspond to an image based on the entropyvalues. In some implementations, the entropy analyzer 430 can segment ascreenshot into blocks having a predetermined size. Image entropy foreach block can be determined, and the likely edges of an image can beinferred based on changes in entropy from block to block. In someimplementations, blocks may be combined into larger regions whoseentropy can be determined by the entropy analyzer 430 in a subsequentround of testing.

In some implementations, the entropy analyzer 430 can determine entropyby applying a function to the distribution of pixel values across thescreenshot. For example, in some implementations, the distribution ofpixel values can be represented by a histogram. In some implementations,a logarithmic function can be applied to the distribution of pixelvalues (e.g., the histogram of the screenshot) to determine entropy. Theentropy analyzer 430 can utilize histogram information to determinewhether an image includes certain colors that may be associated withparticular applications.

In some implementations, the entropy analyzer 430 or the image analyzercan be configured to identify one or more text characters displayedwithin the image. By identifying text characters in the image, the imageanalyzer can identify a brand associated with the image or otherinformation that can be used to identify a reference image, or used bythe screenshot linking system to identify a product associated with theimage.

The watermark detector 435 can be configured to detect the presence of awatermark in an image extracted from a screenshot. As explained ingreater detail below, a watermark may in some implementations beimplemented as a visual or non-visual marker. Thus, in someimplementations, the watermark detector 435 may be referred to as amarker detector. A watermark can be any information configured touniquely identify an image. The watermark may identify, include or embeda GUID. In some implementations, a watermark may be a visual markeradded to an image, such as a QR code or bar code. For example, awatermark may include any type or form of code, such as a string ofcharacters, or a set of images such as emoticons. In someimplementations, a watermark may include one or more strings ofalphanumeric characters together with emoticons or other images. Somewatermarks may include a predetermined character or symbol string at thebeginning or end of the watermark, and may be preceded or followed bystrings of other characters or images that are unique to each watermark.For example, all watermarks may begin with a common set of one or morecharacters or symbols, and may be followed by additional characters orsymbols that are unique to their respective watermarks. In someimplementations, a watermark may be a visual marker associated with animage but not included directly in the image. For example, the watermarkmay be displayed in an area surrounding a perimeter of the image. In onesuch example, the watermark may be included within a heading, a caption,or a comment of the image. Such a watermark may include characters,symbols, or emoticons that are created using a keyboard interface toenter information into a text field associated with the heading,caption, or comment of the image.

In some other implementations, a watermark can include metadataassociated with an image. In still other implementations, a watermarkcan be encoded within data corresponding to the image. For example, awatermark can be generated using an encoding algorithm. The algorithmcan be performed on an initial image I to create a watermarked image I′that is encoded with a message m. In some implementations, the message mcan be a 128-bit message. The algorithm can be selected such thatwatermarked image I′ is imperceptibly different from the initial imageI. In some implementations, uniform random noise in [−3, 3] can be addedto the initial image I. The initial image I, which may be an image inthe RGB color space, can then be converted to the YC_(b)C_(r) colorspace. Both of the chrominance channels in the converted image can beresized to 512×512. Thus, each chrominance channel can be divided into4,096 blocks each having a size of 8×8. Next, a discreet cosinetransform can be applied to each of the 8×8 blocks in I to createI_(DCT). The message m is encoded in each chrominance channel 32 timesaccording to a mapping, 1≤M(i,j)≤4096 ∈

, which takes the i^(th) bit of the j^(th) redundancy and maps to theblock in I_(DCT) in which to encode the bit. Let B_(i,j)=I_(DCT)(M(i,j))be a block that is to be encoded with the i^(th) bit of m. It can beensured that

$\left\lfloor \frac{{Bi},{j\left( {1,1} \right)}}{10} \right\rfloor$is odd if m_(i)=1, and is even otherwise. Then, I_(DCT) can be convertedusing the inverse discrete cosine transform, and the resulting image Ican be resealed to the dimensions of the original image to generate theencoded image I′ can then be converted back to the RGB color space.Thus, I′ is an image in the RGB color space that is visuallyindistinguishable from I, but which includes the encoded message tn.Other algorithms also can be used to encode a watermark message in animage.

In some implementations, a watermark can be added to an image by thecompany, organization, or person who publishes the image. In someimplementations, the watermark may be added to the image before theimage is published. For example, the company, organization, or personwho publishes the image may first alter or modify data corresponding tothe image to include the watermark, prior to publishing the imageincluding the watermark. In some other implementations, the watermarkcan be created after the image has been published. For example, thecompany, organization, or person who publishes the image may edit theimage to include a watermark after the image has been published orotherwise made available for viewing by others. In another example, thewatermark may be created by a third party who was not involved in thepublishing of the image. As discussed above, a watermark may be includedin a comment or caption associated with the image. Typically, commentsand captions may be created after an image has been published. Thus, athird party may post a comment or caption including a watermarkassociated with an image after the image has been published.

The watermark can include information identifying the image as well asinformation relating to a link associated with the image. Including suchinformation directly within a watermark can allow a link associated withthe image to be easily identified, because it may not be necessary tocompare the image or the watermark to a reference image or a referencewatermark. In other implementations, a set of reference watermarks(which also may be referred to as reference markers) may be stored, andeach reference watermark may be associated with a respective link. Thewatermark detector 435 can scan, monitor, parse or otherwise analyze theimage, or an area of the screenshot surrounding a perimeter of theimage, for a known pattern of information corresponding to a watermarkformat. For example, the watermark detector 435 can scan the image orthe peripheral area of the screenshot surrounding the image for aparticular visual marker, such as a code including a string ofcharacters, symbols, emoticons, or other information. In anotherexample, the watermark detector 435 can scan the image for a particularbit pattern in the image data itself in order to extract the watermarkfrom the image. After the watermark detector 435 determines that awatermark is present in an image, the watermark can be compared to theset of reference watermarks. If a match is found, the link correspondingto the matching reference watermark can be identified (for example, bythe screenshot linking system) as a link associated with the watermarkof the image.

The fingerprint generator 440 can be configured to generate afingerprint based on an image. A fingerprint can include any informationuniquely identifying an image. A fingerprint may identify, include orembed a GUID. The fingerprint function may be designed and configured togenerate and/or identify a GUID. In some implementations, a fingerprintmay not be included within an image. Instead, the fingerprint generator440 may process the image data to generate a fingerprint. In someimplementations, the fingerprint can be generated by performing ahashing function on the image data associated with an image. Forexample, because images that differ from one another are represented byunique image data, the fingerprints representing a hash of the imagedata associated with unique images also should be unique. Furthermore,in some implementations, a fingerprint can be a string of bits uniquelyrepresenting the image from which the fingerprint was generated.

In some implementations, the publisher account manager 226 shown in FIG.2A can collect images published by a company and can store the images inthe database 232 of the server 106, also shown in FIG. 2A. The imagescan be processed to generate a reference fingerprint for each image, anda link can be associated with each reference fingerprint. When thefingerprint generator 440 processes an image on the client 102 togenerate a fingerprint, the fingerprint can be transmitted to the server106 and can be compared to the reference fingerprints stored in thedatabase 232. If the server determines that there is a match, the servercan associate a link corresponding to the matched reference fingerprintwith the image.

FIG. 4B is a conceptual diagram depicting an example use case for thesystem of FIG. 2A. As discussed above, the client 102 includes aplurality of applications 204, an image database 210, and the screenshotlinking application 215. The screenshot linking application 215 includesthe watermark detector 435. In this example, the client 102 alsoincludes a screenshot 439. The screenshot 439 includes an image 441,which includes a watermark 443. In some implementations, the screenshot439 can be stored in the image database 210.

The entropy analyzer 430 can be used to identify and/or extract theimage 441 from the screenshot 439, as described in detail above. Afterthe image 441 has been extracted from the screenshot 439, the screenshotlinking application can provide the image 441 to the watermark detector435. The watermark detector 435 can process the image 441 to extract thewatermark 443. As discussed above, in some implementations, thewatermark can provide information relating to a link to be associatedwith the image. Therefore, in this implementation, the link can bedetermined before any communication is sent from the client 102 to theserver 106. In some implementations, a globally unique identifier (GUID)associated with the image, such as the GUID 450, may be sent to theserver 106. For example, the GUID 450 can be stored in the screenshotlinking system 203 so that the screenshot linking system 203 canmaintain a record of the images that have been viewed on the client 102.In some implementations, the GUID 450 can be stored in the database 232by the user account manager 224 shown in FIG. 2A.

FIG. 4C is a conceptual diagram depicting an example use case for thesystem of FIG. 2A. This example is similar to the example shown in FIG.4B, except that the screenshot 439 includes an image 441 without anywatermark. In some implementations, the screenshot 439 can be stored inthe image database 210. The screenshot linking system 203 also includesa fingerprint analyzer 460 and a notification generator 465.

The entropy analyzer 430 can be used to extract the image 441 from thescreenshot 439, as described in detail above. After the image 441 hasbeen extracted from the screenshot 439, the image 441 can be provided tothe fingerprint generator 440. The fingerprint generator 440 can processthe image 441 to generate a fingerprint 451 associated with the image.For example, the fingerprint generator 440 can perform a hashingfunction on the data representing the image 441 to determine afingerprint 451 associated with the image 441. In some implementations,the fingerprint 451 associated with the image may be sent to the server106. For example, the fingerprint 451 can be stored in the screenshotlinking system 203. As discussed above, the server 106 can maintain adatabase of reference fingerprints and associated links. The fingerprintanalyzer 460 can compare the fingerprint 451 to the referencefingerprints. If a match is determined, the link 420 corresponding tothe matched reference fingerprint can be associated with the image. Thenotification generator 465 can generate a notification 415 whichincludes the link 420. The notification can then be transmitted back tothe client 102, so that a user of the client 102 can access the link420.

In some implementations, reference images can be retrieved from socialmedia websites on which content publishers promote products. Forexample, the posts of a particular company can be scanned on a socialmedia site and fingerprints can be generated for each image. The imageand the generated fingerprints can then be saved and can then serve asreference images. Thus, reference fingerprints can be generatedretroactively, after images have been published by a company.

FIG. 4D is a flow chart of a process 470 for analyzing an imageincluding a watermark. In brief overview an image can be extracted froma screenshot (step 472). A watermark can be identified from theextracted image (step 474). A link can be identified from the identifiedwatermark (step 476). A notification can be provided including the link(step 478).

Referring again to FIG. 4D, and in greater detail, the process 470 caninclude extracting an image from a screenshot (step 472). In someimplementations, an image analyzer such as the image analyzer 225 shownin FIG. 4A can be used to extract the image. For example, as discussedabove, a screenshot may contain images as well as other visual elementsthat are associated with the application used to generate thescreenshot. In some implementations, the entropy analyzer 430 candetermine image entropy across the screenshot to determine whichportions of the screenshot correspond to an image. For example, theentropy of the screenshot can provide information relating to theposition of an image within the screenshot, thereby allowing the imageto be extracted from the screenshot. In some implementations, extractingthe image from the screenshot comprises generating a separate image filerepresenting the image that has been extracted from the screenshot.

The process 470 can include identifying a watermark from the extractedimage (step 474). In some implementations, the watermark can beextracted by a watermark detector such as the watermark detector 435shown in FIG. 4A. In some implementations, the watermark can be anyinformation configured to uniquely identify an image. In someimplementations, a watermark may be a visual marker added to an image,such as a QR code or bar code. The watermark also may be or may includeany other form of visual code, such as a string of character, or a setof images such as emoticons. In some implementations, watermarks mayinclude a predetermined character or symbol string at the beginning orend of the watermark, and may be preceded or followed by strings ofother characters or images that are unique to each watermark. Forexample, all watermarks may begin with a common set of two or morecharacters or symbols, and may be followed by additional characters orsymbols that are unique to their respective watermarks. In such anexample, the watermark may be identified by a watermark detector such asthe watermark detector 435 shown in FIG. 4A by scanning the image forthe predetermined set of characters or symbols. After the set ofcharacters or symbols is located, the watermark detector may alsodetermine the additional characters or symbols that follow or precedethe predetermined set of characters as included in the watermark. Insome implementations, a watermark may include a first set of charactersor symbols which may be referred to as a prefix, and a second set ofcharacters or symbols which may be referred to as a suffix. The prefixmay be common to a plurality of watermarks associated with differentimages. That is, the prefix of two or more distinct watermarks may beidentical. In contrast, the suffix may be unique in each watermarkedimage. For example, watermarks may include six characters in total, withthe first two characters corresponding to the common prefix and the lastfour characters corresponding to the unique suffix. Thus, one watermarkcould be represented by the set of characters “*&1234,” where “*&” isthe prefix and “1234” is the suffix. A second watermark associated witha different image could be represented by the set of characters“*&5678,” where the prefix is again “*&,” as it was in the firstwatermark, but the suffix “5678” of the second watermark is differentfrom the suffix of the first watermark. Because the prefix may be commonacross a plurality of watermarks, the watermark detector may first scanan image to locate the common prefix. After the common prefix has beenidentified, the watermark detector may be further configured to scan theremaining four characters to determine the unique suffix of theidentified watermark. In some implementations, the watermark detectorcan include image processing functionality such as optical characterrecognition, to allow the watermark to be located in the image (oraround a perimeter of the image). It will be appreciated by one ofordinary skill in the art that the example discussed above isillustrative only, and that the principles of the above example may beapplied to many variations of types and forms of watermarks withoutdeparting from the scope of this disclosure. For example, in someimplementations the prefix or suffix may have different lengths thanthose discussed above. In addition, in some implementations the suffixmay be common across a plurality of images while the prefix is unique toeach different image. Furthermore, in some implementations the set ofcharacters used to form the prefix and suffix may be different thanthose of the example above. For example, in some implementations theprefix and the suffix may each contain any combination of letters,numbers, special characters, symbols, emoji, icons, or other forms ofgraphical content.

In some implementations, a watermark may be a visual marker associatedwith an image but not included directly in the image. For example, thewatermark may be displayed in an area surrounding a perimeter of theimage. In such example, the watermark may be included within a heading,a caption, or a comment of the image. Thus, the watermark detector mayscan the image as well as an area surrounding a perimeter of the imageto locate a watermark that may include characters, symbols, or emoticonsin a heading, caption, or comment of the image.

In some other implementations, a watermark or marker of an image may notbe visible to a viewer of the image. For example, a watermark caninclude metadata associated with an image but not impacting theappearance of the image. In another example, a watermark can be encodedwithin data corresponding to the image. For example, the leastsignificant bits of each pixel in the image can be used to storeinformation corresponding to a watermark of the image. Thus, datacorresponding to the watermark or marker may change the appearance ofthe image only slightly, but would not be apparent to a human viewingthe image on a display device. In implementations in which the watermarkis not a visual marker, such as those described above, the watermark mayinclude a code embedded in the image data or in metadata associated withthe image. For example, the code may be a binary code, and may includeerror-checking bits. In some implementations, the watermark detector mayread the code and translate the code into another format, such as ahuman readable string of characters that may correspond to thewatermark. The watermark detector can scan the image for a known patternof information corresponding to the watermark format. For example, thewatermark detector can scan the image for a particular visual marker ora particular bit pattern in the image data itself in order to identifythe watermark from the extracted image.

The process 470 can include identifying a link from the identifiedwatermark (step 476). In some implementations, the link can beidentified by a screenshot linking application executing on a clientdevice or by a screenshot linking system remote from the client device.In some implementations, a watermark can be added to an image by thecompany who publishes the image. The watermark can include informationidentifying the image as well as information relating to a linkassociated with the image. Thus, in some implementations, a linkassociated with the image can be easily identified without comparing theimage or the watermark to a reference image or a reference watermark. Inother implementations, a set of reference watermarks may be stored, andeach reference watermark may be associated with a respective link. Theidentified watermark can be compared to the set of reference watermarks.If a match is found, the link corresponding to the matching referencewatermark can be identified as a link associated with the identifiedwatermark. In some implementations, the link can be a URL or a “deeplink” to a particular post within another application that executes onthe client device.

The process 470 can include providing a notification including theidentified link (step 478). In some implementations, the notificationcan be generated on the client device itself. In some otherimplementations, the notification can be generated remotely by ascreenshot linking system and provided to the client device via acommunication interface. The notification can then be displayed to auser of the client device, who can select the link included in thenotification in order to view other content associated with the imagecorresponding to the link. In some implementations, the notification canbe displayed on the client over the active application on the client.For example, if a screenshot is taken while a particular application isactive on the client, the screenshot may be processed as discussed aboveand the notification can be displayed over the active application. Insome implementations, the notification can be a popup notification. Thenotification may include text notifying a user of the client that a linkassociated with the most recently captured screenshot has beenidentified. In some implementations, the link may be displayed in thenotification. In some other implementations, a user can select thenotification to be directed to a different page or application in whichthe associated link is displayed.

FIG. 4E is a flow chart of a process 480 for analyzing an image based ona fingerprint of the image. In brief overview, the process 480 caninclude extracting an image from a screenshot (step 482). A fingerprintof the identified and/or extracted image can be generated (step 484). Amatch can be determined between the fingerprint of the image and afingerprint of a reference image (step 486). A link associated with theimage can be identified, responsive to determining the match (step 488).A notification including the link can be provided (step 490).

Referring again to FIG. 4E, the process 480 can include identifyingwithin and/or extracting an image from a screenshot (step 482). Forexample, as discussed above, a screenshot may contain images as well asother visual elements that are associated with the application used togenerate the screenshot. In some implementations, an image analyzer suchas the image analyzer 225 shown in FIG. 4A can identify or extract theimage from the screenshot. In some implementations, the entropy analyzer430 can determine image entropy across the screenshot to determine whichportions of the screenshot correspond to an image. For example, theentropy of the screenshot can provide information relating to theposition of an image within the screenshot, thereby allowing the imageto be extracted from the screenshot. In some implementations, extractingthe image from the screenshot comprises generating a separate image filerepresenting the image that has been extracted from the screenshot.

The process 480 can include generating a fingerprint of the extractedimage (step 484). In some implementations, the fingerprint can begenerated by a fingerprint generator such as the fingerprint generator440 shown in FIG. 4A. A fingerprint can include any information uniquelyidentifying an image. In some implementations, the fingerprint can begenerated by performing a hashing function on the image data associatedwith an image. In some implementations, a fingerprint can be a string ofbits uniquely representing the image from which the fingerprint wasgenerated.

The process 480 can include determining a match between the fingerprintof the image and a fingerprint of a reference image (step 486). In someimplementations, the match may be determined by a screenshot linkingapplication such as the screenshot linking application 215 executing onthe client 102 as shown in FIG. 4B. In some other implementations, thematch may be determined by a remote screenshot linking system such asthe screenshot linking system 203 shown in FIG. 4B. In someimplementations, images published by a company can be stored in thedatabase. The images can be processed to generate a referencefingerprint for each image, and a link can be associated with eachreference fingerprint. When a fingerprint is generated for the extractedimage, the fingerprint can be compared to the reference fingerprints.

The process 480 can include identifying a link associated with the imageresponse to determining a match (step 488). In some implementations, thelink can be identified by a screenshot linking application executing ona client device or by a screenshot linking system remote from the clientdevice. In some implementations, links corresponding to reference imagescan be stored in a database. Because the fingerprint match determined instep 486 ensures that the image is the same as a reference image, thelink associated with the matched reference image also should beassociated with the original image.

The process 480 can include providing a notification including theidentified link (step 490). In some other implementations, thenotification can be generated remotely by a screenshot linking systemand provided to the client device via a communication interface. In someimplementations, the notification can be generated on the client deviceitself. The notification can then be displayed to a user of the clientdevice, who can select the link included in the notification in order toview other content associated with the image corresponding to the link.

Referring now to FIG. 4F, FIG. 4F is a flow chart of a process 492 foranalyzing an screenshot including an image and a predetermined marker toaccess a resource. In brief overview, a screenshot is accessed fromstorage (step 494). A predetermined marker included in the screenshot isthen detected (step 495). A link to a resource can be identified fromthe identified predetermined marker (step 496). The resource is thenaccessed via the link (step 497).

In further detail, the process 492 can include accessing a screenshotfrom storage (step 494). In some implementations, a screenshot linkingapplication executing on a client device can access screenshots from astorage of the client device. The storage can be shared across multipleapplications executing on the client device. The storage can be a memoryof the client device or any other type of storage component of theclient device. the client device can include one or morecomputer-executable instructions, which when executed on the clientdevice, cause the client device to take a screenshot of the contentdisplayed on a screen of the client device. In some implementations, ascreenshot capturing function of the client device can capture thescreenshot. In some implementations, the client device can capture ascreenshot responsive to receiving predetermined input commands on theclient device. In some implementations, the screenshot can include animage and a predetermined marker. The image can be an image uploaded toa web resource by a third-party, such as an advertiser, a retailer, or auser. The image can be an image uploaded onto a social media website. Insome implementations, the screenshot can also include other areas of theweb resource that surround the image or are otherwise adjacent to theimage. In some implementations, the other areas of the web resource caninclude text characters, icons, visual symbols, icons, or othergraphical components, which can be used by the screenshot linkingsystem. In some implementations, the party uploading the image canupload the image with a predetermined marker that is detectable by thescreenshot linking application. In some implementations, the image canbe embedded with the predetermined marker in such a way that thescreenshot linking application can detect the marker. In someimplementations, the image can be embedded with the predetermined markerin such a way that the screenshot linking application can visuallydetect the marker using image analysis. In some implementations, thepredetermined marker includes a visual marker added to the image that isincluded in the screenshot prior to the screenshot being captured by theclient device.

The process 492 can include detecting a predetermined marker included inthe screenshot (step 495). In some implementations, the screenshotlinking application can detect the predetermined marker included in thescreenshot by extracting the image from the screenshot and thenidentifying the marker from the extracted image (step 474). In someimplementations, the marker can be extracted or otherwise identified bya watermark detector such as the watermark detector 435 shown in FIG.4A. In some implementations, the marker can be any informationconfigured to uniquely identify the image included in the screenshot. Insome implementations, the marker may be a visual marker added to animage, such as a QR code or bar code. The marker also may be or mayinclude any other form of visual code, such as a string of character, ora set of images such as emoticons. In some implementations, markers mayinclude a predetermined character or symbol string at the beginning orend of the marker, and may be preceded or followed by strings of othercharacters or images that are unique to each marker. For example, allmarkers may begin with a common set of two or more characters orsymbols, and may be followed by additional characters or symbols thatare unique to their respective markers. In such an example, the markermay be identified by scanning the image or the screenshot for thepredetermined set of characters or symbols.

The process 492 can include identifying a link to a resource from theidentified predetermined marker (step 496). In some implementations, thelink to the resource can be identified by the screenshot linkingapplication executing on the client device or by a screenshot linkingsystem remote from the client device. In some implementations, thescreenshot linking application or the screenshot linking system canmaintain a plurality of reference markers. The screenshot linkingapplication or the screenshot linking system can maintain a plurality ofreference markers in a marker database that maps the reference markersto at least one resource. In some implementations, the database caninclude a mapping of each reference image to a link to a resource. Inthis way, when the predetermined marker is identified from thescreenshot, the screenshot linking application executing on the clientdevice or the screenshot linking system can determine a match betweenthe predetermined marker and one of the plurality of reference markersstored in the marker database. The screenshot linking application or thescreenshot linking system can analyze the predetermined marker togenerate a hash of the image or some other computed value and comparethe hash or computed value of the predetermined marker to computedvalues of the reference markers. If a match between the computed valueof the predetermined marker and one of the reference markers is found,the screenshot linking application or the screenshot linking systemdetermines that the predetermined marker matches a reference marker. Thescreenshot linking application or the screenshot linking system can thenidentify a link or URL to a resource to which the matched referencemarker is mapped in the marker database. In some implementations, thelink can be a URL or a “deep link” to a particular post within anotherapplication that executes on the client device. The screenshot linkingapplication or the screenshot linking system can then select the link tothe resource that is mapped to the matched reference marker.

The process 492 can include accessing the resource via the link (step497). In some implementations. The client device can access the resourcevia the link responsive to the screenshot linking applicationidentifying the link and sending instructions to the client device (oran application executing on the client device) to access the resourcevia the link. In some implementations, the link can be a URL to aresource in which case, the screenshot linking application can generatea request to cause a browser on the client device to access the resourcevia the URL. In some implementations the link can be a deep link to aspecific application executing on the client device. In some suchimplementations, the screenshot linking application can generate arequest to access the resource to which the link is mapped via thespecific application. In some implementations, the screenshot linkingapplication can generate an actionable object that is displayed on theclient device. The actionable object can be configured to cause theclient device to access the resource to which the link is mappedresponsive to the user of client device taking an action on theactionable object. Examples of actions can include a click, tap, or anyother input that can be received by the client device.

E. Providing a Product Recommendation to a User

Users who capture screenshots depicting particular types of products maynot be interested in other unrelated types of products. However,products similar to those that are depicted in the screenshots that auser captures may be of more interest to the user. As a result, the usermay be more likely to purchase a product that is similar to the productsdepicted in the screenshots captured by the user. In someimplementations, the screenshots and purchasing decisions of the user,as well as those of other users, can form the basis for a productrecommendation for the user.

One aspect of this disclosure can be implemented in a method forproviding a product recommendation to a user. To provide recommendationto one or more users, the screenshot linking system can build one ormore recommendation engines. The recommendation engines can rely onmodels that process input data to generate output data in the form ofrecommendations. The input data can be referred to as training data thatis provided to the recommendation engine to train the model. The modelscan be adaptive and continually learning.

In some implementations, the recommendation engine can be configured todetermine recommendations for a user based on a user's screenshots. Therecommendation engine can determine recommendations of products that aresimilar to products identified in the user's screenshots. In someimplementations, the recommendation engine can rely on screenshots ofother users that the recommendation engine determines are similar toscreenshots captured by the user. In this way, the recommendation enginecan generate product recommendations for the user based on screenshotsand images included in the screenshots captured by the other users ofthe screenshot linking system.

In some implementations, the recommendation engine can be configured toidentify users that are similar to one another based on the screenshotscaptured by each of the users. In this way, the recommendation enginecan recommend products in which users similar to the user have expressedan interest. Similarly, advertisers or other content providers maychoose to generate marketing or advertising campaigns directed towardsusers that are similar to one another and that are likely interested inthe products of the advertisers. The recommendation engine can beconfigured to generate a mode to identify users similar to one anotherbased on the screenshots of each of the users.

The screenshot linking system, by analyzing screenshots of a particularuser, can gain intelligence relating to the user. To gain intelligencerelating to the user, the screenshot linking system can scan the imagedatabase including the screenshots captured by the device of the user.In some implementations, the screenshot linking system can scan theimage database to identify data, including but not limited to, dataindicating the dates and time the screenshot was captured, the number ofscreenshots captured, the frequency with which screenshots have beencaptured, and any geographical location information associated with thescreenshots, among others.

As described above, the screenshot linking system can analyze thescreenshots to identify the identity of applications active on thedevice when the screenshots were captured as well as informationincluded in images included in the screenshots. Based on theapplications identified via the screenshots, the screenshot linkingsystem can determine a manifestation of activity of applications. Thatis, the screenshot linking system can identify the types of applicationsthe user engages with, the frequency of engagement, among others. Insome implementations, the screenshot linking system can determine, fromthe applications identified via the screenshots, activity of the userwith the identified applications.

The screenshot linking system can further monitor activity performed bythe user with one or more stores associated with the screenshot linkingsystem. The screenshot linking system can identify the images includedin the screenshots and based on the identified images, determine auser's purchasing intent. If the screenshot linking system determinesthat the user has captured a larger number of screenshots that includeimages of shoes, the screenshot linking system can determine that theuser may have an intent to purchase shoes. This intent can be used tothen provide recommendations to the user for additional content.Further, the intent can be used by the screenshot linking system toallow advertisers that sell shoes to target the user by providingrelevant content related to shoes.

The screenshot linking system can be configured to identify brandsassociated with the images included in the screenshots. The screenshotlinking system can also identify specific products included in theimages. In some implementations, the screenshot linking system canidentify the specific products and brands by matching the images withreference images, as described above. The screenshot linking system canfurther determine, based on the brands and products identified in theimages, details relating the products, including the price thresholds ofthe products as well as the categories to which the products belong. Forinstance, the screenshot linking system may identify multiple imagesrelating to shoes. The screenshot linking system can further determinespecific shoes and using third-party data or information retrieved froma database mainlined by the screenshot linking system, prices of theshoes. Based on the prices of the shoes and the brands of the shoes, thescreenshot linking system can learn about the user's preferences andshopping habits, as well as be able to learn about products that theuser is or may be interested in.

The screenshot linking system can be configured to generate models tolearn about users as well as to generate a recommendation engine. Thescreenshot linking system can generate user specific models specific torespective users. The model can be configured to determine the types ofproducts a user may be interested in, the price range of the products,and the brands of the products, among others. In some implementations,the screenshot linking system can also generate brand-specific modelsspecific to respective brands. The screenshot linking system canidentify images corresponding to specific brands and identify users thatcaptured screenshots of such images. The screenshot linking system canthen generate a brand-specific model for a brand based on activities ofusers that captured screenshots of images related to the brand.

According to one aspect, a first plurality of user profiles can begenerated for a plurality of users based on screenshots associated withthe plurality of users. A plurality of categories to which the userprofiles belong can be determined. Each user profile can be assigned toat least one of the plurality of categories. A recommendation model canbe built for each category. A second user profile, not included in thefirst plurality of user profiles, can be identified. A category to whichthe second user profile should be assigned can be identified. A productrecommendation can be provided to the second user based on the categoryto which the second user profile is assigned.

According to another aspect, the screenshot linking system can receive aplurality of screenshots captured by a device of a user. The screenshotlinking system can identify images included in the screenshot. Thescreenshot linking system can, from the images, identify one or morecategories to which the images correspond. In some implementations, thecategories can be based on topics, such as clothing, furniture, andelectronics, among others. In some implementations, the categories canbe based on brands. In some implementations, the categories can be basedon other attributes. Responsive to determining the categories that thescreenshots belong, the screenshot linking system can generate a userprofile based on the categories of the screenshots. The screenshotlinking system, via a recommendation engine that uses data from a largenumber of users, can provide recommendations of products (or content ingeneral) based on a comparison of the user's profile with user profilesof other users and the screenshots corresponding to the other users.

Referring now to FIG. 5A, a block diagram depicting an embodiment of acontent recommendation provider 231 that can be used in the system ofFIG. 2A is shown. The content recommendation provider 231 is shown asexecuting on the server 106. However, in other implementations, thecontent recommendation provider 231 can instead execute on the client102. In brief overview, content recommendation provider 231 includes abrand monitor 510, a price monitor 515, a product category monitor 520,a time/location monitor 525, a model builder 530, and a recommendationgenerator 535.

The brand monitor 510 can monitor and record information correspondingto brands associated with images of screenshots that are captured byusers of various client devices, such as the client device 102. Forexample, the brand monitor 510 can determine a brand associated with ascreenshot by referring to the company that published an imageassociated with the screenshot. The brand can be a brand correspondingto a product shown in an image within the screenshot. In someimplementations, the content recommendation provider 231 can be notifiedwhenever a client device, such as the client device 102, sends imagedata to be processed by the screenshot linking system 203. Image datacan include a GUID of a watermark, a fingerprint, an image, or ascreenshot. After the brand monitor 510 determines a brand associatedwith a screenshot, information corresponding to the brand can be storedin a user profile associated with the particular client device on whichthe screenshot was captured. In some implementations, the user accountmanager 224 shown in FIG. 2A can stored the brand information in therespective user profile on the server 106, for example in the database232.

The price monitor 515 can monitor and record information correspondingto prices associated with images of screenshots that are captured byusers of various client devices, such as the client device 102. Forexample, the price monitor 515 can determine a price associated with aproduct shown in a screenshot. In some implementations, the pricemonitor 515 can determine a price range associated with a screenshot.For example, the price range can be a range that includes the actualprice of a product shown in the screenshot. In other implementations,the price monitor 515 can determine a specific price of a product in ascreenshot. After the price monitor 515 determines a price associatedwith a screenshot, information corresponding to the price can be storedin a user profile associated with the particular client device on whichthe screenshot was captured. In some implementations, the user accountmanager 224 shown in FIG. 2A can stored the price information in therespective user profile on the server 106, for example in the database232.

The product category monitor 520 can monitor and record informationcorresponding to product categories associated with images ofscreenshots that are captured by users of various client devices, suchas the client device 102. For example, the product category monitor 520can determine a product category associated with a product shown in ascreenshot. In some implementations, the product category can be a broadcategory, such as “clothing.” In other implementations, the productcategory can be a narrower category, such as “men's clothing” or“women's clothing.” In some implementations, the product categorymonitor 520 can determine more than one category for a screenshot. Afterthe product category monitor 520 determines a product categoryassociated with a screenshot, information corresponding to the productcategory can be stored in a user profile associated with the particularclient device on which the screenshot was captured. In someimplementations, the user account manager 224 shown in FIG. 2A canstored the product category information in the respective user profileon the server 106, for example in the database 232.

The time/location monitor 520 can monitor and record informationcorresponding to times and/or locations associated with screenshots thatare captured by users of various client devices, such as the clientdevice 102. For example, the time/location monitor 520 can determine thetime at which a particular screenshot was captured or the location inwhich the screenshot was captured. In some implementations, the time andlocation information can be determined based on metadata associated witha screenshot. For example, an image file corresponding to a screenshotmay include a time stamp representing the time at which the image filewas created. Similarly, an image file corresponding to a screenshot mayinclude GPS information representing the approximate location of theclient device at the time the image file was created. After thetime/location monitor 520 determines time and location informationassociated with a screenshot, information corresponding to the time andlocation can be stored in a user profile associated with the particularclient device on which the screenshot was captured. In someimplementations, the user account manager 224 shown in FIG. 2A canstored the time and location information in the respective user profileon the server 106, for example in the database 232.

The model builder 530 can generate a model associated with variouscategories. Categories that can be modeled may include brands, prices,product categories, and geographic location categories. In someimplementations, the model builder 530 can build a model based oninformation stored in the user profiles of a plurality of users. Themodel builder 530 can identify trends across users who capturescreenshots of similar products. For example, the model builder 530 candetermine popular products within a particular price range, popularproducts within a particular brand, brands that are popular relative toother brands, and popular product categories. In some implementations,the model builder 530 can generate a model based on more than onecategory. For example, the model builder 530 can determine that, amongusers who capture screenshots of products in a given price range,products from a certain brand are also particularly popular. In someimplementations, the model builder 530 can build an individual model foreach unique user whose client 102 device sends screenshots to the server106. For example, the model for a user may be based in part on theproducts shown in screenshots of other users who appear to have similarproduct preferences. In some implementations, a model may include or maybe based on a generic template. For example, a model template caninclude various attributes and values associated with those attributes.In some implementations, attributes can include brands, products, orproduct categories. Values for each attribute can indicate a weight tobe assigned to that particular attribute. Thus, the generic modeltemplate can be customized according to the preferences of each user ofa client device based on the screenshots captured on the individualclient devices. For example, the values associated with attributes canbe in increased responsive to determining that those attributes are moreprevalent in the screenshots on a client device, and can be decreasedresponsive to determining that those attributes are less prevalent inthe screenshots on a client device.

The recommendation generator 535 can recommend products to users ofclients 102. In some implementations, the recommendation generator 535can use the information provided by the model builder 530 to generate arecommendation. For example, the recommendation generator 535 canreceive information from the model builder 530 corresponding to aproduct that is predicted to be of interest to a particular user, basedon the model for that user. In some implementations, the recommendationgenerator 535 also can generate a notification including arecommendation and can send the notification to the respective client102.

FIG. 5B is an example graphical user interface for displaying productsfound in the screenshots of a client device. The graphical userinterface displays products based on the images extracted fromscreenshots, as discussed above. The user can select the “similar”button at the top of the graphical user interface shown in FIG. 5B to bedirected to the graphical user interface shown in FIG. 5C.

FIG. 5C is an example graphical user interface for displaying productssimilar to those that are found in the screenshots of a client device.In some implementations, the graphical user interface shown in FIG. 5Ccan show products selected based on a model associated with the user ofthe client device on which the graphical user interface is displayed.For example, similar products can be identified as products havingsimilar prices, similar brands, or similar product categories withrespect to the products shown in screenshots of the client device. Suchinformation can be collected by the brand monitor 510, the price monitor515, and the product category monitor 520, and modeled by the modelbuilder 530, as discussed above. In some implementations, similarproducts can be identified by the recommendation generator 535 based ona model built by the model builder 530, and sent to the client device tobe display in the graphical user interface of FIG. 5C. The user canselect the “shots” button at the top of the graphical user interfaceshown in FIG. 5C to be directed to the graphical user interface shown inFIG. 5D. FIG. 5D is a graphical user interface for displaying thescreenshots that are stored on a client device. In some implementations,the screenshots can be retrieved from a database, such as the imagedatabase 210 shown in FIG. 2A. In some implementations, productsdisplayed in the graphical user interface shown in FIG. 5C can beidentified by a recommendation engine, as described further below inconnection with FIG. 5E.

FIG. 5E is a flow chart of a process 560 for providing a productrecommendation to a user. In brief overview, a first plurality of userprofiles can be generated for a plurality of users based on screenshotsassociated with the plurality of users (step 562). A plurality ofcategories to which the user profiles belong can be determined (step564). Each user profile can be assigned to at least one of the pluralityof categories (step 566). A recommendation model can be built for eachcategory (step 568). A second user profile, not included in the firstplurality of user profiles, can be identified, and a category to whichthe second user profile should be assigned can be identified (step 570).A product recommendation can be provided to the second user based on thecategory to which the second user profile is assigned (step 572).

Referring again to FIG. 5E, the process 560 can include generating,using screenshots associated with a plurality of users, a firstplurality of user profiles for the plurality of users (step 562). Insome implementations, the user profiles can be generated by the useraccount manager 224 shown in FIG. 2A. For example, when screenshots orother image data is sent to a screenshot linking system, informationrelating to products shown in the screenshots can be stored in the userprofiles of the users who generated the screenshots. In someimplementations, the information can include information relating tobrands, prices or price ranges, product categories, time stamps, andlocation data associated with products shown in screenshots. As a usercontinues to send additional image data to the screenshot linkingsystem, the user profile for that user can be augmented to includeadditional product information based on the newly received screenshots.

The process 560 can include determining, from the identified userprofiles, a plurality of categories to which the user profiles belong(step 564). In some implementations, the categories may be determined bythe product category monitor 520 shown in FIG. 5A. Categories mayinclude brands, prices, or product categories. Each user profile can beassigned to at least one of the plurality of categories (step 566). Insome implementations, a user account manager can assign the userprofiles to respective categories. In some implementations, a userprofile can be assigned to a category based on the information in theuser profile. For example, if a user profile indicates that the useroften captures screenshots that show sneakers, the user profile can beadded to a “sneakers” category. In some implementations, the userprofile also can be added to other categories. The other categories mayoverlap with the first category, and may be broader or narrower than thefirst category. For example, a user profile that is added to a“sneakers” category also may be added to a broader “footwear” category.User profiles can also be added to other categories based on the brandsand prices of products that appear in their screenshots.

The process 560 can include building a recommendation model for eachcategory (step 568). In some implementations, a model builder can buildthe model for each category. For example, models can be built based oninformation stored in the user profiles of a plurality of users. Themodel builder can identify trends across users who capture screenshotsthat belong to a particular category. For example, the model builder candetermine popular products within a particular price range, popularproducts within a particular brand, brands that are popular relative toother brands, and popular product categories. In some implementations,the model builder 530 can generate a model based on more than onecategory. For example, the model builder 530 can determine that, amongusers who are assigned to a particular price category, products from acertain brand are also particularly popular.

The model builder 530 can be configured to generate a model using afirst set of known users. The model builder 530 may receive, as inputs,information associated with a plurality of users. The information caninclude the screenshots captured by each of the users, their demographicinformation, their preferences, likes and dislikes, and other dataeither inferred from the users or received from the users. In addition,the model builder may receive category information associated with eachuser. Category information can include information that categorizes theuser in one or more categories, for instance, brands, type of clothing,type of fashion, among others. The model builder can then analyze eachof the screenshots of each of the users to determine trends across eachof the users based on their screenshots and their known categories,preferences, likes and dislikes.

The process 560 can include identifying, for a second user profile notincluded in the first plurality of user profiles, a category to whichthe second user profile should be assigned (step 570). In someimplementations, the category for the second user may be determined bythe product category monitor 520 shown in FIG. 5A. In someimplementations, the category of the second user profile can bedetermined based on the products shown in screenshots that have beencaptured by the second user. For example, the screenshots of the seconduser can be analyzed to determine a category that the second user islikely to be interested in, and the second user can then be assigned tothat category. In some implementations, the second user may be assignedto more than one category.

In some implementations, the model builder can identify the seconduser's screenshots and other profile information. Based on the imagesincluded in the screenshots and the profile information, the modelbuilder can match the second user to one or more of the first usersbased on similarities of the images and the profile information. Forinstance, if the images in the screenshots of the second user aresimilar to images included in screenshots of a subset of the firstplurality of users, the second user can be determined to be similar tothat subset of the first plurality of users. In some implementations,the model builder can generate feature vectors for each of the usersbased on their preferences and information extracted from the images,and using algorithms, determine clusters of users based on distances ofthe feature vectors from one another.

The process 560 can include providing a product recommendation to thesecond user based on the category to which the second user profile isassigned (step 572). In some implementations, the product recommendationcan be provided by the recommendation generator 535 shown in FIG. 5A. Insome implementations, the product recommendation can be selected basedon a model associated with the category to which the second user isassigned. For example, the second user can be assigned to an “athleticwear” category. The model for that category may indicate that usersassigned to that category are often interested in particular brands,such as Nike. As a result, a recommendation for a Nike product may beprovided to the second user, even if the second user has never captureda screenshot showing a Nike product. In some implementations, therecommendation can be provided on the client device of the second userthrough a graphical user interface similar to the graphical userinterface discussed above in connection with FIG. 5C.

In other implementations, the screenshot linking system can receive aplurality of screenshots captured by a device of a user. The screenshotlinking system can identify images included in the screenshots. Thescreenshot linking system can, from the images, identify one or morecategories to which the images correspond. In some implementations, thecategories can be based on topics, such as clothing, furniture, orelectronics, among others. In some implementations, the categories canbe based on brands. In some implementations, the categories can be basedon other attributes. Responsive to determining the categories to whichthe screenshots belong, the screenshot linking system can generate auser profile based on the categories of the screenshots. The screenshotlinking system, via the recommendation engine that uses data from alarge number of users, can provide recommendations of products (orcontent in general) based on a comparison of the user's profile withuser profiles of other users and the screenshots corresponding to theother users.

In some implementations, each image can be categorized with one or moretags. These tags can be provided based on information received from thepublisher of the image. In some implementations, the tags can begenerated using third-party sources. In some implementations, the tagscan be generated from information received from users of the screenshotlinking system. In some implementations, the screenshot linking systemcan analyze each of the images of a user and determine the tags orattributes associated with each of the images. The screenshot linkingsystem can then generate a user profile based on the attributes. Theuser profile can be a multi-attribute vector that represents the user.The vector can then be used to categorize the user in one or morecategories. In some implementations, the screenshot linking system candetermine a strength of each attribute based on a number of times theattribute is tagged to the images included in the user's screenshots.Examples of attributes can include product items, product color, productshape, product price, product brand, among others. It should beappreciated that the images can include products corresponding to a widevariety of industries. Although the examples shown herein relate toclothing, the products can include furniture products, electronics,kitchenware, home goods, cars, among others. In addition, each of thedifferent types of products may have different appropriate attributes.For instance, for cars, the type of car can be an SUV, van, sedan, orconvertible, among others, while for furniture, the type of furniturecan include a sofa, table, chair, dining table, or ottoman, amongothers. As such, attributes may vary based on the type of productidentified in an image. As such, in some implementations, therecommendation engine may only generate recommendations for a user basedon images corresponding to particular products, for instance, cars,furniture, clothing, or subsets of such product types.

In some implementations, the screenshot linking system may be able todetermine a user's conversions responsive to links provided to the user.The conversions can influence the recommendation engine of thescreenshot linking system as conversions can provide stronger signals ofpreference than screenshots. Using the vector representing the user, therecommendation engine can identify one or more products or content thatthe recommendation determines to be relevant to the user. The identifiedproducts or content are then provided to the user and responsive to theuser's actions on the products or content, the recommendation engine canupdate the strengths of attributes of the user.

In some implementations, the screenshot linking system can determinetrends based on multi-attribute vectors representing the users. Forinstance, if the multi-attribute vectors of many users indicate a strongattribute in a particular brand, the recommendation engine can determinethat the particular brand is popular. Similarly, if the multi-attributevectors of many users indicate a strong attribute in a particular color,the recommendation engine can determine that the particular color ispopular. While the invention has been particularly shown and describedwith reference to specific embodiments, it should be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the inventiondescribed in this disclosure.

In some implementations, the process 560 can include different oradditional steps, and the steps shown in FIG. 5E also may be modified,in order to provide a product recommendation to a user. For example, insome implementations, the recommendation generator can use variousmachine learning techniques to provide a recommendation. In someimplementations, the recommendation generator can adapt the models usedto provide recommendations based on feedback from a user. Thus,subsequent recommendations to the user can be improved, based on theusers responses to past recommendations (e.g., whether the userexpressed interest or made purchases of products recommended in thepast). In some implementations, the recommendation generator can providepersonalized recommendations to a user by adjusting the models used togenerate recommendations based on past behavior of the user.

What is claimed is:
 1. A computer-implemented method for processingimage data using at least one processor coupled to a memory, the methodcomprising: receiving, by the at least one processor, a set of imagedata generated from a display device at a client computing deviceassociated with a user; determining, by the at least one processor, thatthe received set of image data represents a first screenshot based onmetadata associated with the received set of image data; identifying, bythe at least one processor, a first plurality of visual elements of thefirst screenshot; determining, by the at least one processor, firstrelevant content in the first screenshot based on the identified firstplurality of visual elements; matching, by the at least one processor,the first relevant content in the first screenshot with first referencecontent; identifying, by the at least one processor, a first productcategory or a first product brand to which the first relevant contentcorresponds; determining, by the at least one processor, a first linkassociated with the first reference content; transmitting, by the atleast one processor, the first link to the client computing device toenable the user to access the first link; generating, by the at leastone processor, a user profile for the user to include data associatedwith the first relevant content in the first screenshot; andassociating, by the at least one processor, the user profile with theidentified first product category.
 2. The method of claim 1, wherein thefirst relevant content is determined in the first screenshot using imageprocessing techniques.
 3. The method of claim 1, wherein the first linkprovided to the client computing device is a URL to a product webpagefor purchasing a product identified in the screenshot.
 4. The method ofclaim 1, wherein the first relevant content includes a marker configuredto uniquely identify the first relevant content.
 5. The method of claim1, further comprising identifying at least one application associatedwith the first screenshot based on the identified first plurality ofvisual elements.
 6. The method of claim 1, further comprising:monitoring, by the at least one processor, a database associated withthe client computing device to determine when new sets of image datathat correspond to screenshots are added to the database.
 7. The methodof claim 1, further comprising generating and transmitting anotification to the client computing device that the first linkassociated with the first screenshot has been determined.
 8. The methodof claim 1, wherein the first relevant content is an image in the firstscreenshot, and wherein matching the first relevant content in the firstscreenshot with the first reference content comprises: comparing theimage to a plurality of reference images stored within the memory; anddetermining a match between the image and one of the plurality ofreference images.
 9. The method of claim 1, wherein the first linkcomprises a deep link corresponding to at least one of a particularwebpage or a portion of a particular mobile application associated withthe first screenshot.
 10. The method of claim 1, further comprising:identifying, by the at least one processor, a second plurality of visualelements in a second screenshot; determining, by the at least oneprocessor, second relevant content in the second screenshot based on thesecond plurality of visual elements; matching, by the at least oneprocessor, the second relevant content in the second screenshot with asecond reference content; identifying, by the at least one processor, asecond product category to which the second relevant content in thesecond screenshot corresponds, wherein the second product category isthe same as the first product category; determining, by the at least oneprocessor, a second link associated with the second reference content;transmitting, by the at least one processor, the second link to theclient computing device to enable the user to access the second link;and updating the user profile associated with the first product categoryto include data associated with the second relevant content in thesecond screenshot.
 11. A system for analyzing image data comprising: atleast one processor; and at least one storage device storinginstructions which, when executed by the at least one processor, causethe at least one processor to perform operations comprising: receiving aset of image data generated from a display device at a client computingdevice associated with a user; determining that the received set ofimage data represents a first screenshot based on metadata associatedwith the received set of image data; identifying a first plurality ofvisual elements of the first screenshot; determining first relevantcontent in the first screenshot based on the identified first pluralityof visual elements; matching the first relevant content in the firstscreenshot with first reference content; identifying a first productcategory or a first product brand to which the first relevant contentcorresponds; determining a first link associated with the firstreference content; transmitting the first link to the client computingdevice to enable the user to access the first link; generating, by theat least one processor, a user profile for the user to include dataassociated with the first relevant content in the first screenshot; andassociating, by the at least one processor, the user profile with theidentified first product category.
 12. The system of claim 11, whereinthe first relevant content is determined in the first screenshot usingimage processing techniques.
 13. The system of claim 11, wherein thefirst relevant content includes a marker configured to uniquely identifythe first relevant content.
 14. The system of claim 11, wherein theoperations further comprise monitoring a database associated with theclient computing device to determine when new sets of image data areadded to the database.
 15. The system of claim 11, wherein theoperations further comprise generating and transmitting a notificationto the client computing device that the first link associated with thefirst screenshot has been determined.
 16. The system of claim 11,wherein the first link comprises a deep link corresponding to at leastone of a particular webpage or a portion of a particular mobileapplication associated with the first screenshot.
 17. A non-transitorycomputer-readable medium storing computer-executable instructions which,when executed by at least one processor, cause the at least oneprocessor to perform operations comprising: receiving a set of imagedata generated from a display device at a client computing deviceassociated with a user; determining that the received set of image datarepresents a first screenshot based on metadata associated with thereceived set of image data; identifying a first plurality of visualelements of the first screenshot; determining first relevant content inthe first screenshot based on the identified first plurality of visualelements; matching the first relevant content in the first screenshotwith first reference content; identifying a first product category or afirst product brand to which the first relevant content corresponds;determining a first link associated with the first reference content;transmitting the first link to the client computing device to enable theuser to access the first link; generating, by the at least oneprocessor, a user profile for the user to include data associated withthe first relevant content in the first screenshot; and associating, bythe at least one processor, the user profile with the identified firstproduct category.
 18. The computer-readable medium of claim 17, whereinthe first link provided to the client computing device is a URL to aproduct webpage for purchasing a product identified in the firstscreenshot.
 19. The computer-readable medium of claim 17, wherein thefirst relevant content includes a marker configured to uniquely identifythe first relevant content.
 20. The computer-readable medium of claim17, wherein the first link comprises a deep link corresponding to atleast one of a particular webpage or a portion of a particular mobileapplication associated with the first screenshot.